Distributed multi-path market information processing method and system

Abstract

The invention discloses a distributed multi-path market information processing method and system, and relates to the field of data processing, and the method comprises the steps: receiving multi-path market information data; selecting one path of market information data with the minimum time delay as optimal market information data; basic information of optimal market data is added; according to the basic information, obtaining supplementary information corresponding to the security code by querying a security database; according to the supplementary information and the optimal market data containing the basic information, generating security static data; generating corresponding K-line data and derivative data according to the optimal market data containing the basic information; subscribing optimal market information data, K-line data and derivative data containing basic information from the message queue service by using stream subscription through the market information pushing service; aiming at the problem that the time delay of multi-path market data processing is large in the prior art, the method receives the multi-path market data through a plurality of access points, selects the path with the minimum time delay as the optimal market data, and reduces the processing time delay of the market data.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a distributed multi-channel market information processing method and system. Background Art

[0002] With the rapid development of financial markets and the increasing popularity of quantitative trading, traditional market systems have been unable to meet the real-time, stability and data diversity requirements of quantitative strategy trading systems. Quantitative trading systems need to process massive amounts of historical market data for strategy backtesting, and at the same time require the market system to provide real-time market data to support real-time calculation and order placement of strategies. In addition, quantitative trading systems also need to calculate various aggregation and derivative indicators of market data, which places higher requirements on the data processing capabilities and response speed of the market system.

[0003] Most of the existing market information systems adopt a centralized architecture, which centralizes the market information of various exchanges and data suppliers to a single market information server for processing. This architecture has many shortcomings: first, a single market information server is difficult to cope with the high concurrent requests of the quantitative trading system, and it is easy to become the performance bottleneck of the system; second, the centralized architecture lacks flexible data integration and expansion capabilities, and it is difficult to quickly introduce new market information sources and data types; third, a single point failure of the market information server may cause the entire trading system to paralyze, posing a greater system risk.

[0004] In addition, the existing market system also has deficiencies in data processing and distribution. Different market sources provide data in different formats, lacking unified data specifications and standardized processing, resulting in poor data readability and usability. The aggregated data such as K-line and time-sharing generated by the market system and various derivative indicators often cannot meet the personalized needs of quantitative trading systems. The transmission delay of market data is large, making it difficult to support the real-time requirements of high-frequency trading strategies. Summary of the invention

[0005] In response to the problem of long delay in processing multi-channel market data in the prior art, the present application provides a distributed multi-channel market data processing method and system, which receives multi-channel market data through multiple access points, selects the channel with the smallest delay as the optimal market data, etc., thereby reducing the processing delay of market data.

[0006] The purpose of this application is achieved through the following technical solutions.

[0007] One aspect of the present application provides a distributed multi-channel market data processing method, including: S1, receiving multi-channel market data through multiple access points; S2, selecting a channel of market data with the smallest delay as the optimal market data, and accessing the selected optimal market data to a message queue; obtaining the optimal market data containing the basic information by adding the basic information of the optimal market data; S3, obtaining supplementary information corresponding to the securities code by querying a securities database based on the basic information; generating securities static data based on the supplementary information and the optimal market data containing the basic information; S4, generating corresponding K-line data and derivative data through data fitting based on the optimal market data containing the basic information; S5, using streaming subscription through a market push service to subscribe to the optimal market data, K-line data and derivative data containing the basic information from a message queue service; S6, receiving a client request through an interface gateway service, and forwarding the client request to a query service.

[0008] Among them, the Market Data Push Service is a service component specifically used to push market data to clients in real time. It obtains the latest market data through interaction with the message queue service, and pushes this data to clients that have subscribed to market information as soon as possible, thus achieving real-time updates of market data. Market data push services usually use some high-performance, low-latency communication protocols, such as WebSocket or TCP long connections, to ensure the timeliness and stability of data transmission. At the same time, the market data push service can also provide customized data push solutions according to the different needs of the client, such as subscription by securities code, subscription by market type, etc.

[0009] Streaming Subscription is a data transmission and consumption mode, characterized by data transmission in the form of continuous, real-time data streams, and data consumers (subscribers) can dynamically subscribe or unsubscribe from data streams and consume data in real time according to their own processing capabilities. Compared with traditional data subscription modes (such as timed polling), the advantages of streaming subscription are higher real-time data transmission and greater flexibility in data consumption, which can meet the characteristics of rapid updates and dynamic changes in market data. Common streaming subscription modes include publish-subscribe mode, push mode, etc.

[0010] Message Queue Service is a middleware service used to implement asynchronous communication between different software components. It achieves decoupling between different components by passing messages (such as market data) from message producers to message consumers, thereby improving the scalability and fault tolerance of the system. Common message queue services include Apache RabbitMQ, ActiveMQ, etc. In the above-mentioned distributed multi-channel market processing method, the message queue service acts as a bridge connecting market data producers (such as market data access modules) and market data consumers (such as market push services), realizing efficient transmission and sharing of market data between different modules. At the same time, the message queue service also provides some advanced functions, such as message persistence, message filtering, message partitioning, etc., which further enhance the reliability and performance of the system.

[0011] Furthermore, basic information includes security code, security name and exchange code; supplementary information includes security type and trading status; and derivative data reflects security price trends and trading activity.

[0012] Further, S2, selects a channel of market data with the smallest delay as the optimal market data, and connects the selected optimal market data to the message queue, including: S21, stores the received multiple channels of market data in the order of arrival time in the specified topic of the RabbitMQ stream processing engine, and associates a timestamp with each market data, the timestamp indicates the arrival time of the market data; S22, uses the time window mechanism of the RabbitMQ stream processing engine to group the market data in the specified time window according to the securities code, and obtains the market data stream corresponding to each securities code; S23, according to the market data stream corresponding to each securities code, uses the double-ended queue data structure to set a fixed-size sliding window, and stores the data in the sliding window. timestamp of the market data; S24, when new market data arrives, determine whether the timestamp of the market data is greater than the maximum timestamp in the sliding window, if so, add the timestamp of the new market data to the end of the sliding window, and delete the timestamp at the head of the sliding window; S25, obtain the market data corresponding to the minimum timestamp in the sliding window, as the market data with the smallest delay for the corresponding security code in the current sliding window, as the optimal market data; S26, connect the optimal market data to the RabbitMQ message queue through the RabbitMQ producer API, and publish the optimal market data to the specified topic of the RabbitMQ stream processing engine in the form of a key-value pair; the key in the key-value pair is the security code, and the value is the optimal market data.

[0013] Among them, Topic, in RabbitMQ, is a concept similar to a database table or a file system directory, which is used to store and manage a group of related message data. Each topic can be partitioned to achieve parallel processing and load balancing of data. Different types of message data can be published to different topics to achieve logical isolation and orderly processing of data. In the above market processing method, by publishing the market data of different securities codes to different topics, efficient storage and management of market data can be achieved, which is convenient for subsequent data processing and analysis.

[0014] The time window mechanism is a commonly used data processing mechanism in the field of stream processing, which is used to aggregate, calculate or analyze data within a certain time range. By dividing the continuous data stream into different windows according to the time dimension, the data in each window can be treated as a whole for processing, thereby achieving the time sensitivity and real-time performance of data processing. Common time window types include tumbling windows, sliding windows, and session windows. In the above market processing method, by utilizing the time window mechanism of RabbitMQ, the market data within the specified time range can be grouped according to the securities code, and the market data stream corresponding to each securities code can be obtained, which provides a data basis for the subsequent optimal market selection.

[0015] Producer API, in RabbitMQ, the Producer API is a programming interface for publishing message data to RabbitMQ topics. Through the Producer API, the application can serialize the message data into a byte array, specify the metadata information of the message (such as topic, partition, key, etc.), and then send the message to the RabbitMQ cluster. RabbitMQ provides producer APIs in multiple programming languages, such as Java, Python, Go, etc., to facilitate message publishing in different application scenarios and development environments. In the above market processing method, by using the RabbitMQ producer API, the selected optimal market data can be published to the specified topic of RabbitMQ in the form of key-value pairs for subsequent data consumption and processing.

[0016] Further, S3 generates securities static data based on the supplementary information and the optimal market data containing the basic information, including: S31, extracting the securities code from the optimal market data containing the basic information as a query condition, and querying the supplementary information corresponding to the securities code in the securities database through a predefined database query interface; S32, according to predefined data encapsulation rules, associating the supplementary information with the optimal market data containing the basic information to generate securities static data; parsing the securities static data into the following Go language data structure, the data structure includes Code for the securities code, Name for the securities name, Exchange for the exchange code, Type for the securities type, and Status for the transaction status.

[0017] S33, serialize the securities static data using the Protobuf data serialization format, the serialization steps are as follows: generate the corresponding Protobuf message definition file (.proto) according to the Go language structure of the securities static data; use the Protobuf compiler to compile the .proto file into a Go language code file; reference the generated code in the Go code; create the Protobuf message object corresponding to the securities static data and fill in the fields; call the serialization method of the Protobuf message object to generate binary serialized data; publish the serialized securities static data to the specified topic of the RabbitMQ stream processing engine, with the securities code as the routing key (Routing Key) and the serialized binary data as the message body; S34, synchronize the securities static data in the RabbitMQ stream processing engine to the distributed column storage database, with the securities code as the distinguishing key and the various fields of the securities static data as columns; S35, when the supplementary information in the securities database changes, synchronize the change information to the RabbitMQ stream processing engine through the Binlog-based database change mechanism.

[0018] Further, S4, based on the optimal market data containing basic information, generates corresponding K-line data and derivative data through data fitting, including: S41, obtaining the optimal market data containing basic information from the specified topic of the RabbitMQ stream processing engine, and grouping and sorting the optimal market data according to the securities code and timestamp to obtain the time series market data corresponding to each securities code; S42, performing K-line synthesis on the time series market data corresponding to each securities code to generate corresponding K-line data; S43, calculating the technical indicators of the time series market data corresponding to each securities code, and generating corresponding derivative data; S44, associating the generated K-line data and derivative data according to the securities code and timestamp, and publishing the associated data to the specified topic of the RabbitMQ stream processing engine; S45, synchronizing the K-line data and derivative data in the RabbitMQ stream processing engine to the distributed time series database, with the securities code as the partition key, the timestamp as the sorting key, and the various fields of the K-line data and derivative data as measurement indicators.

[0019] Further, S42, K-line synthesis is performed on the time series market data corresponding to each securities code to generate corresponding K-line data, including: according to a preset K-line period, the time window size of the K-line synthesis is set, and the time series market data is divided according to the time window size to obtain a continuous time window sequence; the indicators of each time window sequence are calculated to generate the K-line data of the corresponding time window; the indicators include opening price, closing price, highest price, lowest price and trading volume; the K-line data is denoised by using the 3σ principle to obtain denoised K-line data; according to the denoised K-line data, the price change rate and the trading volume change rate between adjacent K-line data points are calculated to obtain a trend feature sequence of the K-line data; according to the trend feature sequence, run-length encoding is used to compress the data to obtain the final K-line data.

[0020] Furthermore, the opening price is the price of the first market data in the time window; the closing price is the price of the last market data in the time window; the highest price is the maximum value of all market data prices in the time window; the lowest price is the minimum value of all market data prices in the time window; and the trading volume is the sum of the trading volumes of all market data in the time window.

[0021] Further, S43, calculating the technical indicators of the time series market data corresponding to each securities code, generating corresponding derivative data, including: calculating the moving average indicator according to the K-line data, the moving average indicator includes , , , and ;in, The calculation formula is: ,in, represents the closing price of the i-th day, and n represents the number of cycles of the moving average; the exponential moving average indicator is calculated based on the time series market data and the K-line data. The exponential moving average indicator includes and ;in, Calculated by the following recursive formula: ,in, , n represents the period of the exponential moving average, Indicates the closing price of the current trading day. Indicates the value of the n-day exponential moving average of the previous trading day; and Calculate the MACD indicator, which includes DIF, DEA and MACD; wherein DIF, DEA and MACD are calculated by the following formula: , , ,in, The 9-day exponential moving average of DIF is represented; the volume index is calculated based on the time series market data, and the volume index includes VOL and AMT; wherein VOL and AMT are calculated by the following formula: , ,in, Indicates the transaction volume of the i-th market data, represents the price of the ith market data; the volume ratio index VR is calculated according to VOL; the VR is calculated by the following formula: ,in, Indicates the transaction volume of the current trading day. The 5-day simple moving average of the trading volume; the price change rate indicator is calculated based on the time series market data, and the price change rate indicator includes and ;in, and Calculated by the following formula: , ,in, Indicates the closing price of the current trading day. Indicates the closing price of the previous trading day. Indicates the closing price 5 trading days ago; based on the volume change rate indicator and the price change rate indicator Calculating Volume Price Trend Indicator ; Calculated by the following formula: , , where n represents the number of calculation cycles, Indicates the transaction volume of the current trading day. Indicates the trading volume n trading days ago; the calculated , , MACD, VOL, AMT, VR, and As the derivative data, it is associated with the corresponding securities code and timestamp, and the associated derivative data is arranged in the order of the securities code and timestamp to generate the final derivative data result.

[0022] Furthermore, it also includes: S6, receiving the client request through the interface gateway service, and forwarding the client request to the query service; the query service queries the corresponding securities static data from the securities database according to the client request, and obtains the corresponding K-line data and derivative data from the market push service; the securities static data, K-line data and derivative data are combined and packaged to obtain the packaged client data, and the packaged client data is returned to the interface gateway service; the interface gateway service sets the response time threshold of the client request. For the client request that exceeds the response time threshold, the interface gateway service terminates the processing of the corresponding client request and returns the request timeout prompt information to the client.

[0023] Distributed service fault handling based on interface monitoring and dynamic expansion includes: S61, using a distributed tracking system to track and monitor the request call links of each service through the interface monitoring service, collecting service request volume, response time, error rate and other indicator data, and using a log aggregation platform to collect service operation logs, extracting error information and exception stacks in the logs; S62, the interface monitoring service uses a stream processing engine to perform real-time calculation and rule matching on the indicator data collected in step S61. The stream processing engine processes the indicator data in the form of an event stream in the memory based on the memory computing technology, and judges the service through predefined aggregation functions and anomaly detection algorithms. Whether the running status meets the preset normal status threshold, and the calculation results are output to the downstream system in the form of a stream; S63, if the running status of the service does not meet the normal status threshold, the interface monitoring service uses the service orchestration engine to automatically generate a fault handling workflow according to the preset fault handling strategy. The fault handling workflow includes a series of fault diagnosis, recovery operations and compensation operations. The workflow engine coordinates the calls and executions between different services to achieve self-healing and recovery of the service; at the same time, the interface monitoring service uses an intelligent alarm system to perform semantic analysis and correlation analysis on the fault handling information, automatically generates fault reports and handling suggestions, and sends them to the operation and maintenance personnel through the multi-channel alarm platform.

[0024] S64, the interface monitoring service uses a time series database to perform persistent storage and aggregation calculations on the indicator data collected in step S61. The time series database uses column storage and compression technology to support efficient data writing and query. Through predefined aggregation functions and window functions, sliding window calculations and multi-dimensional analysis are performed on the indicator data to determine whether the load status of the service exceeds the preset load threshold, and the calculation results are displayed in the form of a time series on the monitoring screen.

[0025] S65, if the load status of the service exceeds the load threshold, the interface monitoring service uses an adaptive expansion algorithm to dynamically calculate the optimal number of service instances according to the preset expansion strategy. The adaptive expansion algorithm is based on reinforcement learning technology. Through the feedback of historical load data and expansion effects, it continuously optimizes the expansion strategy and parameters to achieve intelligent elastic scaling of the service. At the same time, the interface monitoring service uses shadow deployment technology to verify and preheat the expanded service instances without affecting online business, ensuring smooth expansion and stable operation of the service.

[0026] S66, the dynamic expansion mechanism creates a distributed copy instance of the service through the container orchestration platform, and registers the newly created service instance to the service registration center. At the same time, it updates the routing rules of the interface gateway service; S67, the interface gateway service uses a load balancing algorithm to dynamically distribute client requests to different service instances based on the updated routing rules and the service instance information provided by the service registration center. The load balancing algorithm includes polling, random, minimum number of connections, weighted, etc.; S68, after the expanded service instance processes the request, it returns the processing result to the interface gateway service. The interface gateway service integrates the processing result and returns it to the client. At the same time, it reports the performance indicators of the request processing to the interface monitoring service; the interface monitoring service dynamically adjusts the number of service instances according to the performance indicators and the load status of the service to achieve elastic scaling of the service.

[0027] Another aspect of the present application further provides a distributed multi-channel market information processing system for executing a distributed multi-channel market information processing method of the present application.

[0028] Compared with the prior art, the advantages of this application are:

[0029] By receiving multiple channels of market data at multiple access points, using the time window mechanism and sliding window algorithm of the RabbitMQ stream processing engine, the market data delays of different access points are compared in real time, and the channel with the smallest delay is selected as the optimal market data. Compared with the existing technology, this solution avoids invalid processing of market data with large delay, thereby significantly reducing the overall processing delay of market data.

[0030] After selecting the optimal market data, this solution obtains basic information and supplementary information related to the market data from the securities database through data enhancement processing to generate complete securities static data. At the same time, the data fitting algorithm is used to generate corresponding K-line data and derivative data based on the optimal market data. Compared with the existing technology, this solution fully explores the potential value of market data and maximizes the utilization rate of market data through data association and derivative indicator calculation.

[0031] For securities static data, this solution uses distributed column storage to support efficient storage and fast query of massive data. For K-line data and derivative data, this solution uses a distributed time series database to support high-compression storage and fast range query of data. At the same time, the streaming subscription mechanism of the RabbitMQ message queue is used to achieve real-time push of market data, K-line data, and derivative data. Compared with the existing technology, this solution meets the real-time requirements of market data while ensuring the efficiency of data storage and query.

[0032] This solution introduces interface gateway service and query service to uniformly access and route client requests. By setting the request timeout threshold, the system avalanche caused by request accumulation can be prevented. At the same time, the interface monitoring service obtains the operating status of each service in real time, sets the service load threshold, combines fault handling and dynamic expansion mechanism, and automatically creates service instances according to the load situation, thereby ensuring the high availability of the system. In addition, this solution adopts microservice architecture and containerized deployment, which can achieve elastic expansion and rapid deployment of services and has good scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The present application will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0034] Figure 1 It is an exemplary flow chart of a method for processing distributed multi-channel market information according to some embodiments of the present application. DETAILED DESCRIPTION

[0035] The method and system provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0036] like Figure 1As shown, a distributed multi-channel market processing method includes: receiving multi-channel market data through multiple access points; selecting a channel of market data with the smallest delay as the optimal market data, and connecting the selected optimal market data to the message queue; obtaining the optimal market data containing the basic information by adding the basic information of the optimal market data; obtaining the supplementary information corresponding to the securities code by querying the securities database according to the basic information; generating securities static data according to the supplementary information and the optimal market data containing the basic information; generating corresponding K-line data and derivative data according to the optimal market data containing the basic information through data fitting; subscribing to the optimal market data, K-line data and derivative data containing the basic information from the message queue service by using streaming subscription through the market push service; receiving the client request through the interface gateway service, and forwarding the client request to the query service.

[0037] Specifically, S1 receives multiple market data through multiple access points; the system sets up multiple market data access points, such as the Shanghai Stock Exchange, Shenzhen Stock Exchange, and Hong Kong Stock Exchange. Each access point is connected to the market release system of the exchange through a dedicated line or VPN, and receives Level-1, Level-2 and other market data released by each exchange in real time. Taking the Shanghai Stock Exchange Level-1 market as an example, real-time market data is received through the TCP protocol. The data format is binary, including fields such as security code, latest price, trading volume, buy price, buy volume, sell price, sell volume, etc.

[0038] S2, select the market data with the smallest delay as the optimal market data, and connect the selected optimal market data to the message queue, including: S21, store the received multiple market data in the specified topic of the RabbitMQ stream processing engine in the order of arrival time, and associate a timestamp with each market data, and the timestamp indicates the arrival time of the market data. In specific implementation, the source connector (SourceConnector) of RabbitMQ Connect can be used to write the market data of different exchanges into different topics (Topic) of the RabbitMQ cluster in real time. Each topic corresponds to the market data stream of an exchange. The data format can be a structured format such as JSON, protobuf, and include fields such as securities code, price, and trading volume. When writing to RabbitMQ, the market data of the same securities code can be assigned to the same partition (Partition) through a custom partitioner (Partitioner) to ensure the orderliness of the data. At the same time, a timestamp field can be added to each market data to record the arrival time of the data. The timestamp can be the time when the market is released or the time when the market is received. The time accuracy is recommended to reach the millisecond level.

[0039] S22, using the time window mechanism of the RabbitMQ stream processing engine, the market data within the specified time window is grouped according to the security code to obtain the market data stream corresponding to each security code. In specific implementation, the time window API (such as Time Windows, Session Windows, etc.) provided by RabbitMQ Streams DSL can be used to group and aggregate the market data within a certain time range according to the security code. The length of the time window can be set according to the update frequency and real-time requirements of the market data, and can generally be set between 100 milliseconds and 1 second. After grouping and aggregation, each security code has a corresponding KStream or KTable, which represents the market data stream or status table of the security.

[0040] S23, according to the market data stream corresponding to each securities code, a fixed-size sliding window is set using a double-ended queue data structure, and the timestamp of the market data is stored in the sliding window. In the RabbitMQ Streams application, a Key Value Store is defined using the Stores API to store the timestamp queue corresponding to each securities code. The Key of the Key Value Store is the securities code, and the Value is a double-ended queue of timestamps. In the setup method of the RabbitMQ Streams application, the Key Value Store is initialized, and an empty double-ended queue is created for each securities code. In the transform method of the RabbitMQ Streams application, each piece of market data is processed, its timestamp is extracted, and the double-ended queue of the corresponding securities code is updated. In the transform method of the RabbitMQ Streams application, according to the timestamp of the current market data, the minimum timestamp in the double-ended queue is queried as the basis for selecting the optimal market data. The RabbitMQ Streams application can maintain a fixed-size sliding window for each securities code, and the timestamps of the most recent N pieces of market data are stored in the window. The size of the sliding window can be adjusted according to the update frequency and real-time requirements of the market data. The larger the window, the longer the time span, and the higher the delay of the selected optimal market data may be; the smaller the window, the shorter the time span, and the lower the delay of the selected optimal market data may be, but the jitter and error of the data may also be greater. In this application, the double-ended queue supports insertion and deletion operations at the head and tail of the queue, with a time complexity of O (1), and no other elements need to be moved, which is very efficient. The double-ended queue only stores timestamps, not complete market data, has a small memory footprint, and can support concurrent processing of a large number of securities codes. Using a double-ended queue to implement a sliding window can significantly reduce the delay in selecting market data, improve the timeliness of the optimal data, and reduce memory and CPU overhead.

[0041] S24, when new market data arrives, determine whether the timestamp of the market data is greater than the maximum timestamp in the sliding window. If so, add the timestamp of the new market data to the end of the sliding window and delete the head timestamp of the sliding window. In specific implementation, the market data stream of each securities code can be processed in the RabbitMQ Streams application, and each data can be customized using the transform or process method. In the processing function, first obtain the timestamp of the data and compare it with the maximum timestamp in the sliding window. If the timestamp of the new data is greater than the maximum timestamp, add the new timestamp to the end of the sliding window and delete the head timestamp.

[0042] S25, obtain the market data corresponding to the minimum timestamp in the sliding window, as the market data with the minimum delay of the corresponding securities code in the current sliding window, as the optimal market data. In specific implementation, the market data stream of each securities code can be processed in the RabbitMQ Streams application, and each data can be customized using the transform or process method. In the processing function, the minimum timestamp in the sliding window is obtained, and then the corresponding market data is queried from the state storage (such as Key Value Store) according to the minimum timestamp, and the query result is output as the optimal market data.

[0043] S26, connect the optimal market data to the RabbitMQ message queue through the RabbitMQ producer API, and publish the optimal market data to the specified topic of the RabbitMQ stream processing engine in the form of a key-value pair; the key in the key-value pair is the securities code, and the value is the optimal market data. In specific implementation, the optimal market data for each securities code can be output to the specified RabbitMQ topic using the to method in the RabbitMQ Streams application. The output data format can adopt a structured format such as JSON, protobuf, etc., and the securities code is used as the key of the message, and the optimal market data is used as the value of the message. Subsequent consumer applications can partition and consume and process the optimal market data according to the securities code.

[0044] In this application, RabbitMQ Connect is responsible for real-time data collection and publishing, RabbitMQ Streams is responsible for real-time data calculation and aggregation, and double-ended queues and sliding windows are responsible for data caching and latency control. Compared with the original market data, the optimal market data can significantly reduce data transmission and storage overhead, and reduce network bandwidth and disk space usage. At the same time, through sliding window and timestamp comparison, the data with the lowest latency can be dynamically selected from multiple market data to improve the real-time and accuracy of the data. In addition, RabbitMQ Streams provides highly scalable and fault-tolerant stream processing capabilities, which can easily cope with the concurrent processing of massive market data, and supports fast state recovery and replay.

[0045] The basic information included in the optimal market data is as follows: Securities code: a unique identifier of a stock or fund in the form of "600000.SH". Securities name: the abbreviation of a stock or fund such as "Shanghai Pudong Development Bank". Exchange code: the exchange to which the securities belong in the form of "SH", "SZ", etc. By querying the securities basic database, you can obtain supplementary information of the securities corresponding to the optimal market data, such as: Securities type: stocks, funds, bonds, etc. Trading status: normal trading, suspension, delisting, etc. Based on the optimal market data, you can further calculate derivative data to reflect the price trend and trading activity of the securities, such as: Price increase or decrease: the percentage increase or decrease of the current price relative to the previous closing price. Amplitude: the ratio of the difference between the highest and lowest prices of the day to the previous closing price. Turnover rate: the ratio of trading volume to outstanding shares. Through multi-point access to obtain redundant multi-channel market data, the time window mechanism and sliding window algorithm of the stream processing engine are used to dynamically select the market data with the smallest delay as the optimal data, and then combined with the message queue to achieve low-latency and highly reliable market data distribution. At the same time, the content of market data is enriched through data enhancement and derivative indicator calculation.

[0046] S3, based on the supplementary information and the optimal market data containing basic information, generates securities static data, including: S31, extracts the securities code from the optimal market data to query supplementary information: The optimal market data usually contains basic information such as securities code, price, trading volume, etc., which can be parsed using JSON, Protobuf and other formats. After parsing, the securities code field can be extracted using regular expressions, string matching and other methods. After extracting the securities code, you can construct an SQL statement or call the ORM framework to query the securities database through a predefined database query interface (such as a DB connection pool). The return result of the query interface is generally the supplementary information data corresponding to the securities code, such as the securities name, exchange, securities type, etc.

[0047] S32, Supplementary information and basic information are associated to generate static data: The supplementary information queried and the original basic information are assembled and formatted to generate securities static data. The association here can be performed in memory, such as using a HashMap with the securities code as the key and the static data as the value. For the convenience of subsequent processing, the static data is generally parsed into a structure / class of a specific programming language, and the Go language is used here.

[0048] S33, Protobuf serializes static data and publishes it: First, define an equivalent Protobuf message based on the Go structure of the static data and generate a .proto file. Then use the protoc tool to compile the .proto file into the corresponding Go source code file, which mainly contains the message structure definition and serialization method. In the code, create a Protobuf message object and assign the values ​​of each field of the static data to the message object. Call the serialization method of the Protobuf message to serialize the object into binary data. Use the securities code as the routing key and the serialized binary data as the message body, and publish it to the specified topic of RabbitMQ. When publishing, you can use the Go language client library of RabbitMQ, such as amqp.

[0049] S34, static data storage: Subscribe to the static data topic in RabbitMQ to obtain the newly published message. Call the deserialization method of Protobuf to parse the binary message body into a static data structure. Then write the static data to a columnar database, such as HBase, Cassandra, etc. When writing, use the stock code as the primary key, and store each field in a separate column, which can be done with the help of the ORM framework.

[0050] S35, incremental data synchronization: Configure the Binlog of the securities database and write the changed data to the Binlog file in real time. Develop a Binlog parser to read the changed content of the Binlog file in real time. Parse the changed data into a static data structure and publish it to a specific topic in RabbitMQ. The downstream subscribes to the changed data from the topic and updates the static information in the database.

[0051] S4, according to the optimal market data containing basic information, generate corresponding K-line data and derivative data through data fitting, including: S41, obtain the optimal market data containing basic information from the specified topic of the RabbitMQ stream processing engine, and group and sort the optimal market data according to the securities code and timestamp to obtain the time series market data corresponding to each securities code. For example, using the KStream and KGrouped Stream API of RabbitMQ Streams, read the optimal market data from the topic named "optimal", group them according to the securities code, and sort them according to the timestamp to obtain the time series market data corresponding to each securities code. By grouping and sorting by securities code, the scattered market data can be converted into a continuous time series, which is convenient for subsequent K-line synthesis and indicator calculation.

[0052] S42, perform K-line synthesis on the time series market data corresponding to each securities code to generate corresponding K-line data; according to the preset K-line cycle, set the time window size of K-line synthesis, and divide the time series market data according to the time window size to obtain a continuous time window sequence; calculate the index of each time window sequence to generate the K-line data of the corresponding time window; the index includes opening price, closing price, highest price, lowest price and trading volume; the opening price is the price of the first market data in the time window; the closing price is the price of the last market data in the time window; the highest price is the maximum value of all market data prices in the time window; the lowest price is the minimum value of all market data prices in the time window; the trading volume is the sum of the trading volumes of all market data in the time window. For example, if the preset K-line cycle is 1 minute, the time series market data is divided according to the 1-minute time window, and each time window contains multiple market data. Then, the market data in each time window is calculated by the index to generate the corresponding K-line data. By setting a fixed-size time window, market data of different frequencies can be converted into unified K-line data, which is convenient for subsequent data analysis and visualization.

[0053] The 3σ principle is used to denoise the K-line data to obtain the denoised K-line data. For example, for the market data in each time window, its price mean μ and standard deviation σ are calculated, and the outliers beyond the range of [μ-3σ, μ+3σ] are removed, and the K-line indicators are recalculated to obtain the denoised K-line data. Denoising can eliminate the sharp fluctuations and erroneous data in the market data and improve the quality and reliability of the K-line data.

[0054] According to the denoised K-line data, by calculating the price change rate and trading volume change rate between adjacent K-line data points, the trend feature sequence of the K-line data is obtained. For example, for two adjacent K-line data points k1 and k2, their price change rate and trading volume change rate are:

[0055] , ,in, Indicates K-line data points The closing price of Indicates K-line data points Closing price. Indicates K-line data points The transaction volume, Indicates K-line data points of trading volume.

[0056] Arrange the price change rate and volume change rate of each K-line data point in chronological order to obtain the trend feature sequence of the K-line data, which reflects the changing trend of securities prices and trading activity. According to the trend feature sequence, run-length encoding is used for data compression to obtain the final K-line data. Run-length encoding is a run-length encoding algorithm that can compress consecutive data points of the same value into the form of (value, length), greatly reducing the storage space of the data. For example, for the trend feature sequence [0.01, 0.01, 0.02, -0.01, -0.01], it can be compressed into the form of [(0.01, 2), (0.02, 1), (-0.01, 2)]. The use of run-length encoding can reduce the storage and transmission overhead of data while retaining data characteristics, thereby improving the scalability of the system.

[0057] This application can generate K-line data in real time from massive market data, and improve the efficiency and quality of data processing through optimization methods such as data denoising, feature extraction and compression coding. This streaming computing architecture can make full use of RabbitMQ's message caching and horizontal expansion capabilities to achieve low-latency generation and real-time push of K-line data. At the same time, through multi-level caching and indexing of K-line data, it can support high-concurrency queries of clients, avoid direct access to the underlying time series database, and significantly reduce query latency.

[0058] S43, calculate the technical indicators of the time series market data corresponding to each securities code, generate corresponding derivative data, and calculate the moving average indicators based on the K-line data, including , , , and .by Taking the calculation as an example, the closing price of the last n trading days corresponding to each securities code is read from the K-line data topic of RabbitMQ. According to the formula Calculate The value of Indicates the closing price of the i-th day. The calculation result is output to the derived data topic of RabbitMQ in the form of (symbol, timestamp, value), such as ("600000.SH", "2022-03-01 09:31:00", 10.25).

[0059] Calculate the exponential moving average indicator based on time series market data and K-line data, including and .by For example, the closing price of the most recent trading day for each stock code is read from the RabbitMQ market data topic. , read the previous trading day's data from the derived data topic value , according to the recursive formula Calculate the current trading day Value, where , n takes values ​​of 12 and 26. The calculation results are output to the Derived Data topic in the same format as MA(n).

[0060] according to and Calculates MACD indicators, including DIF, DEA, and MACD. Reads the latest data for each security code from the Derivative Data topic. and The values ​​of DIF, DEA and MACD are calculated according to the following formula: , , ,in, Represents the 9-day exponential moving average of DIF. The calculation result is output to the derived data topic in the form of (symbol, timestamp, dif, dea, macd).

[0061] Calculate the volume indicators, including VOL and AMT, based on the time series market data. Read all the market data of the most recent trading day corresponding to each security code from the market data topic, and calculate the values ​​of VOL and AMT according to the following formula: , ,in, Indicates the transaction volume of the i-th market data, Indicates the price of the i-th market data. The calculation result is output to the derived data topic in the form of (symbol, timestamp, vol, amt).

[0062] Calculate the volume ratio indicator VR based on VOL. Read the VOL value of the last five trading days corresponding to each security code from the derivative data topic, calculate the simple moving average MA (VOL, 5) of the trading volume in these five days, and then use the formula Calculate the VR value of the current trading day, where Indicates the transaction volume of the current trading day. The calculation result is output to the derived data topic in the form of (symbol, timestamp, vr).

[0063] Calculates price change rate indicators based on time series market data, including and Read the closing price of the last six trading days for each security code from the market data topic and calculate it according to the following formula and Values: , ,in, Indicates the closing price of the current trading day. Indicates the closing price of the previous trading day. Indicates the closing price 5 trading days ago. The calculation result is output to the derived data topic in the form of (symbol, timestamp, change1, change5).

[0064] According to the volume change rate indicator and the price change rate indicator Calculating Volume Price Trend Indicator .by For example, the VOL value of the last five trading days and the closing price of the last six trading days corresponding to each security code are read from the derivative data topic, and the following formula is used to calculate and Values: ; ;in, Indicates the transaction volume of the current trading day. Indicates the trading volume 5 trading days ago. The calculation of is detailed in the above formula. The calculation result is output to the derived data topic in the form of (symbol, timestamp, vppi5).

[0065] The calculated , , MACD, VOL, AMT, VR, and Derivative data such as securities codes and timestamps are associated with the corresponding securities codes and timestamps, and arranged in the order of securities codes and timestamps to generate the final derivative data results. This application adopts the state storage and sliding window functions of RabbitMQ Streams to achieve incremental calculation and real-time update of derivative data. Compared with traditional batch processing methods, this solution can significantly reduce computing latency and improve the real-time nature of data. At the same time, by dividing different derivative indicators into different RabbitMQ partitions, parallel computing can be achieved to further improve processing performance.

[0066] S44, associate the generated K-line data and derivative data according to the securities code and timestamp, and publish the associated data to the specified topic of the RabbitMQ stream processing engine. Specifically, read the latest data corresponding to each securities code from the K-line data topic and the derivative data topic of RabbitMQ, respectively, perform the connection operation according to the timestamp, obtain the result data containing the complete K-line and derivative data fields, and publish the result data in the protobuf format to the result data topic of RabbitMQ, and use the securities code as the key of the message to ensure the orderly distribution and consumption of data with the same securities code.

[0067] By associating K-line data and derived data in RabbitMQ, the complexity of data acquisition and calculation for downstream data consumers can be reduced, and data availability can be improved. The application of protobuf format can achieve efficient serialization and deserialization of data, reducing data transmission and storage overhead. The orderly distribution of result data helps to ensure the timeliness and consistency of data, and provides support for real-time calculation and display.

[0068] S45, synchronizes the K-line data and derived data in the RabbitMQ stream processing engine to the distributed time series database, using the stock code as the partition key, the timestamp as the sort key, and the various fields of the K-line data and derived data as the measurement indicators. Specifically, use the Sink Connector provided by the time series database (such as InfluxDB, Prometheus) to write the data in the RabbitMQ result data topic to the time series database in real time. In the database, use the stock code as the Tag, the timestamp as the Time, and the various fields of the K-line data and derived data (such as the opening price, closing price, trading volume, MA, MACD, etc.) as the Field to build a suitable data model and index structure to support efficient storage and query of massive time series data.

[0069] By storing data in a time series database, you can make full use of the high-throughput writing, aggregate computing, multi-dimensional query and other features of the time series database to achieve persistent storage and flexible analysis of data. Time series databases usually use columnar storage and compression technology, which can significantly reduce data storage costs and query latency. At the same time, time series databases provide a rich query language and visualization tools that can support complex data analysis and monitoring alarm scenarios, providing comprehensive data support for quantitative trading.

[0070] S5, receives client requests through the interface gateway service and forwards the client requests to the query service; the client sends a RESTful-style data query request to the interface gateway service through the HTTP / HTTPS protocol, and the request parameters include the securities code, time range, data type (such as K-line data, derivative data), etc. The interface gateway service uses a high-performance Web server (such as Nginx, Envoy) to process client requests, and supports concurrent processing and load balancing of requests. Through preset routing rules and service discovery mechanisms (such as Consul, Eureka), the interface gateway service dynamically routes requests to the backend query service cluster.

[0071] The query service adopts a microservice architecture and consists of multiple independent service instances, each of which is responsible for processing data queries for a portion of securities codes. The query service consumes securities static data from the RabbitMQ result data topic in real time and caches the data in local memory or distributed cache (such as Redis) to increase the speed of data query. For K-line data and derivative data, the query service communicates with the market push service through the gRPC protocol to obtain data within the specified securities code and time range in real time. The market push service is built on RabbitMQ Streams and can support real-time data push and query.

[0072] The query service combines and encapsulates the static data of securities, K-line data and derivative data in a predefined data format, and generates response data in JSON or Protocol Buffers format that meets the client's request. The query service returns the encapsulated response data to the interface gateway service, which compresses and encrypts the response data (such as gzip, SSL) and transmits the response data to the client.

[0073] The interface gateway service sets a response time threshold (such as 500 milliseconds) for client requests. For requests that exceed the threshold, the interface gateway service will actively terminate the request processing and return a request timeout prompt to the client to avoid long-term occupation of server resources and long waiting of the client.

[0074] As the traffic entrance of the system, the application interface gateway service can perform unified authentication, authorization, current limiting, and fuse processing on requests to improve the security and stability of the system. The query service adopts a microservice architecture and caching mechanism to support large-scale data queries and real-time responses. The market push service uses RabbitMQ's message caching and stream processing capabilities to achieve real-time data push and on-demand acquisition. Data is transmitted between services using an efficient binary protocol, reducing the serialization and deserialization overhead of data. At the same time, by setting a request timeout mechanism, infinite waiting for requests and resource exhaustion can be prevented, thereby improving the availability of the system.

[0075] S6, this embodiment also includes distributed service fault handling based on interface monitoring and dynamic expansion, S61, through the interface monitoring service, a distributed tracking system is used to track and monitor the request call link of each service, collect the service request volume, response time, error rate and other indicator data, and use the log aggregation platform to collect the service operation log, extract the error information and exception stack in the log. For example, using Zipkin as a distributed tracking system, sampling the request of each service, recording the start time, end time, service name, Span ID, Trace ID and other information of the request, and sending the tracking data to the Zipkin server for storage and analysis. Using ELK (Elasticsearch, Logstash, Kibana) as a log aggregation platform, the operation logs of each service are collected through Logstash, and the log data is transmitted to Elasticsearch for indexing and storage, and the log data is visualized through Kibana. Query and analysis.

[0076] S62, the interface monitoring service uses a stream processing engine to perform real-time calculation and rule matching on the indicator data collected in step S61. The stream processing engine processes the indicator data in the form of an event stream in memory based on in-memory computing technology, and determines whether the running status of the service meets the preset normal status threshold through predefined aggregation functions and anomaly detection algorithms, and outputs the calculation results to the downstream system in the form of a stream. For example, using Flink as a stream processing engine, the indicator data collected by Zipkin and ELK are sent to Flink in the form of RabbitMQ messages, and anomaly detection rules are defined through Flink's CEP (Complex Event Processing) library, such as request response time exceeding 500ms, request error rate exceeding 1%, etc., and events that match the rules are output to the alarm system or storage system.

[0077] S63, if the running status of the service does not meet the normal status threshold, the interface monitoring service uses the service orchestration engine to automatically generate a fault handling workflow according to the preset fault handling strategy. The fault handling workflow includes a series of fault diagnosis, recovery operations and compensation operations. The workflow engine coordinates the calls and executions between different services to achieve self-healing and recovery of the service. At the same time, the interface monitoring service uses the intelligent alarm system to perform semantic analysis and association analysis on the fault handling information, automatically generate fault reports and handling suggestions, and send them to the operation and maintenance personnel through the multi-channel alarm platform. For example, using Airflow as a service orchestration engine, define a fault handling workflow DAG (Directed Acyclic Graph), including fault diagnosis nodes, restart service nodes, rollback version nodes, data recovery nodes, etc. When receiving an abnormal event output by Flink, the execution of the workflow is automatically triggered. Using natural language processing and machine learning algorithms, semantic analysis and association analysis are performed on logs, indicators, events and other information in the fault handling process, and information such as fault causes, impact ranges, and recovery plans are automatically generated, and the operation and maintenance personnel are notified through multiple channels such as emails, SMS, WeChat, and phone calls.

[0078] S64, the interface monitoring service uses a time series database to perform persistent storage and aggregate calculations on the indicator data collected in step S61. The time series database uses column storage and compression technology to support efficient data writing and querying. Through predefined aggregation functions and window functions, the indicator data is subjected to sliding window calculations and multi-dimensional analysis to determine whether the load status of the service exceeds the preset load threshold, and the calculation results are displayed in the form of a time series on the monitoring screen. For example, using InfluxDB as a time series database, the indicator data collected by Zipkin and ELK is written into InfluxDB, and aggregate calculations are performed through InfluxQL query statements, such as calculating the average response time per minute, the number of requests per hour, etc. The query results of InfluxDB are displayed through Grafana, and the load status and change trend of the service are presented in the form of intuitive charts.

[0079] S65, if the load state of the service exceeds the load threshold, the interface monitoring service uses an adaptive expansion algorithm to dynamically calculate the optimal number of service instances according to the preset expansion strategy. The adaptive expansion algorithm is based on reinforcement learning technology. Through the feedback of historical load data and expansion effects, the expansion strategy and parameters are continuously optimized to achieve intelligent elastic scaling of the service; at the same time, the interface monitoring service uses shadow deployment technology to verify and preheat the expanded service instance without affecting the online business, ensuring smooth expansion and stable operation of the service. For example, the Deep Q-learning algorithm is used as an adaptive expansion algorithm, with the service's CPU utilization, memory utilization, request volume and other indicators as the state, and the number of expanded or reduced instances as the action, and the reward value is obtained by interacting with the environment to learn the best expansion strategy. Before expanding the new service instance, the shadow deployment technology is first used to import a small amount of real traffic into the new instance, and the performance, function, compatibility, etc. of the new instance are verified. If the verification passes, the new instance is added to the production environment to replace or supplement the original service instance.

[0080] S66, the dynamic expansion mechanism creates distributed copy instances of the service through the container orchestration platform, registers the newly created service instances with the service registry, and updates the routing rules of the interface gateway service. For example, using Kubernetes as a container orchestration platform, you can automatically increase or decrease the number of Pods of a service by modifying the replicas property of the service's Deployment, or by using HPA (Horizontal Pod Autoscaler). Using Eureka or Consul as a service registry, register the newly created service instance information with the registry for other services to discover and call. Using Zuul or Kong as an interface gateway service, you can modify the routing rules to distribute the request traffic to the new service instance according to a certain strategy.

[0081] S67, the interface gateway service uses a load balancing algorithm to dynamically distribute client requests to different service instances based on the updated routing rules and the service instance information provided by the service registry. The load balancing algorithm includes polling, random, minimum number of connections, weighted, etc. For example, using Ribbon as a client load balancer, through integration with Eureka, obtain the list of service instances in the service registry, and select an instance for request forwarding according to different load balancing strategies. Common strategies include simple polling, weighted polling, random, minimum number of connections, etc. You can choose the appropriate strategy according to the characteristics and needs of the service, such as using the minimum number of connections strategy for CPU-sensitive services and the weighted polling strategy for IO-sensitive services.

[0082] S68, after the expanded service instance processes the request, it returns the processing result to the interface gateway service, which integrates the processing result and returns it to the client. At the same time, it reports the performance indicators of the request processing to the interface monitoring service; the interface monitoring service dynamically adjusts the number of service instances according to the performance indicators and the load status of the service to achieve elastic scaling of the service. For example, the expanded service instance returns the request processing result to the Zuul gateway through HTTP or RPC protocol, and the Zuul gateway aggregates, filters, converts and other operations on the processing results of multiple instances, and returns them to the client through HTTP response. At the same time, the Zuul gateway asynchronously sends the performance indicators such as the processing time, response status code, and exception information of each request to Zipkin or ELK, which are aggregated and analyzed by the interface monitoring service. When the interface monitoring service finds that the load indicator of the service has returned to the normal level and remains stable for a period of time, it will trigger the adaptive expansion algorithm, calculate the optimal number of service instances, and notify the Kubernetes orchestration platform to reduce the number of Pods of the service, release excess computing resources, and achieve elastic contraction of the service.

Claims

1. A method for processing distributed multi-channel market information, characterized in that: include: S1, receiving multiple market data through multiple access points; S2, select the market data with the smallest delay as the optimal market data, and add the selected optimal market data to the message queue; By adding the basic information of the optimal market data, the optimal market data including the basic information is obtained; S3, based on the basic information, obtaining supplementary information corresponding to the securities code by querying the securities database; Generate static data of securities based on supplementary information and optimal market data containing basic information; S4, generating corresponding K-line data and derivative data through data fitting based on the optimal market data containing basic information; S5, using streaming subscription through the market push service, subscribes to the best market data, K-line data and derivative data containing basic information from the message queue service; S6, receiving the client request through the interface gateway service and forwarding the client request to the query service.

2. The method for processing distributed multi-channel market information according to claim 1, characterized in that: Basic information includes security code, security name and exchange code; Supplementary information includes security type and trading status; Derivative data reflect securities price trends and trading activity.

3. The method for processing distributed multi-channel market information according to claim 2, characterized in that: S2, select the market data with the smallest delay as the optimal market data, including: S21, storing the received multiple market data in the designated topic of the RabbitMQ stream processing engine in the order of arrival time, and associating a timestamp with each market data, where the timestamp indicates the arrival time of the market data; S22, using the time window mechanism of the RabbitMQ stream processing engine, grouping the market data within the specified time window according to the securities code, and obtaining the market data stream corresponding to each securities code; S23, according to the market data stream corresponding to each securities code, using a double-ended queue data structure to set a sliding window of a fixed size, wherein the timestamp of the market data is stored in the sliding window; S24, when new market data arrives, determine whether the timestamp of the market data is greater than the maximum timestamp in the sliding window, and if so, add the timestamp of the new market data to the end of the sliding window and delete the head timestamp of the sliding window; S25, obtaining the market data corresponding to the minimum timestamp in the sliding window as the market data with the minimum delay for the corresponding security code in the current sliding window, and as the optimal market data; S26, connect the optimal market data to the RabbitMQ message queue through the RabbitMQ producer API, and publish the optimal market data to the specified topic of the RabbitMQ stream processing engine in the form of a key-value pair; the key in the key-value pair is the securities code, and the value is the optimal market data.

4. The method for processing distributed multi-channel market information according to claim 2, characterized in that: S3, based on the supplementary information and the optimal market data containing basic information, generates securities static data, including: S31, extracting the securities code from the optimal market data containing the basic information as a query condition, and querying the securities database for supplementary information corresponding to the securities code through a predefined database query interface; S32, according to the predefined data encapsulation rules, the supplementary information is associated with the optimal market data containing the basic information to generate securities static data; the securities static data is parsed into a data structure of the GO language, wherein the data structure of the GO language includes a securities code, a securities name, an exchange code, a securities type and a transaction status; S33, serializes securities static data using protobuf data serialization format; S34, publishing the serialized static securities data to the specified topic of the RabbitMQ stream processing engine, using the securities code as the routing key and the serialized binary data as the message body; S35, synchronize the securities static data in the RabbitMQ stream processing engine to the distributed column storage data, using the securities code as the distinguishing key and the various fields of the securities static data as columns; When the supplementary information in the securities database changes, the change information is synchronized to the RabbitMQ stream processing engine through the Binlog-based database change mechanism.

5. The method for processing distributed multi-channel market information according to claim 4, characterized in that: S4, generates corresponding K-line data and derivative data, including: S41, obtaining optimal market data containing basic information from a specified topic of the RabbitMQ stream processing engine, and grouping and sorting the optimal market data according to the securities code and timestamp to obtain time series market data corresponding to each securities code; S42, performing K-line synthesis on the time series market data corresponding to each security code to generate corresponding K-line data; S43, calculating the technical indicators of the time series market data corresponding to each security code and generating corresponding derivative data; S44, associating the generated K-line data and the derivative data according to the securities code and the timestamp, and publishing the associated data to a designated topic of the RabbitMQ stream processing engine; S45, synchronizes the K-line data and derived data in the RabbitMQ stream processing engine to the distributed time series database, using the security code as the partition key, the timestamp as the sort key, and the various fields of the K-line data and derived data as measurement indicators.

6. The method for processing distributed multi-channel market information according to claim 5, characterized in that: S42, performing K-line synthesis on the time series market data corresponding to each security code to generate corresponding K-line data, including: According to the preset K-line cycle, set the time window size of K-line synthesis, and divide the time series market data according to the time window size to obtain a continuous time window sequence; Calculate the indicators of each time window sequence and generate K-line data of the corresponding time window; the indicators include opening price, closing price, highest price, lowest price and trading volume; The 3σ principle is used to denoise the K-line data to obtain the denoised K-line data; According to the denoised K-line data, the trend characteristic sequence of the K-line data is obtained by calculating the price change rate and transaction volume change rate between adjacent K-line data points; According to the trend feature sequence, run-length encoding is used to compress the data to obtain the final K-line data.

7. The method for processing distributed multi-channel market information according to claim 5, characterized in that: The opening price is the price of the first market data in the time window; The closing price is the price of the last market data in the time window; The highest price is the maximum value of all market data prices within the time window; The lowest price is the minimum price of all market data within the time window; The trading volume is the sum of all market data trading volumes within the time window.

8. The method for processing distributed multi-channel market information according to claim 5, characterized in that: S43, calculating the technical indicators of the time series market data corresponding to each securities code, and generating corresponding derivative data, including: The moving average indicator is calculated based on the K-line data. The moving average indicator includes , , , and ;in, The calculation formula is: ; in, represents the closing price of the i-th day, and n represents the period of the moving average; The exponential moving average indicator is calculated based on the time series market data and K-line data. The exponential moving average indicator includes and ;in, Calculated by the following recursive formula: ; in, , n represents the period of the exponential moving average, Indicates the closing price of the current trading day. Indicates the value of the n-day exponential moving average of the previous trading day; according to and Calculate the MACD indicator, which includes DIF, DEA and MACD; wherein DIF, DEA and MACD are calculated by the following formula: ; ; ; in, Represents the 9-day exponential moving average of DIF; The trading volume index is calculated according to the time series market data, and the trading volume index includes VOL and AMT; wherein VOL and AMT are calculated by the following formula: ; ; in, Indicates the transaction volume of the i-th market data, Indicates the price of the i-th market data; The volume ratio index VR is calculated based on VOL; the VR is calculated by the following formula: ; in, Indicates the transaction volume of the current trading day. The 5-day simple moving average representing volume; The price change rate index is calculated based on the time series market data. The price change rate index includes and ;in, and Calculated by the following formula: ; ; in, Indicates the closing price of the current trading day. Indicates the closing price of the previous trading day. Indicates the closing price 5 trading days ago; According to the volume change rate indicator and the price change rate indicator Calculating Volume Price Trend Indicator ; Calculated by the following formula: ; ; Where n represents the number of calculation cycles, Indicates the transaction volume of the current trading day. Indicates the trading volume n trading days ago; The calculated , , MACD, VOL, AMT, VR, and As the derivative data, it is associated with the corresponding securities code and timestamp, and the associated derivative data is arranged in the order of the securities code and timestamp to generate the final derivative data result.

9. The method for processing distributed multi-channel market information according to any one of claims 2 to 8, characterized in that: Also includes: S6, receiving the client request through the interface gateway service and forwarding the client request to the query service; The query service queries the corresponding securities static data from the securities database according to the client's request, and obtains the corresponding K-line data and derivative data from the market push service; Combine and encapsulate securities static data, K-line data and derivative data to obtain encapsulated client data, and return the encapsulated client data to the interface gateway service; The interface gateway service sets a response time threshold for client requests. For client requests that exceed the response time threshold, the interface gateway service terminates the processing of the corresponding client request and returns a request timeout prompt to the client.

10. A distributed multi-channel market information processing system, characterized in that: include: At least one processing unit; used to execute instructions to implement the distributed multi-channel market processing method described in any one of claims 1 to 9.

Images