System for realizing transaction detection for real-time quotation of stock based on asynchronous event
The system addresses real-time and scalability limitations in synchronous polling by using asynchronous events and Redis for data transfer, ensuring high availability and performance in detecting abnormal stock market movements.
Patent Information
- Application Number
- CN202411192420.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-12-06
AI Technical Summary
Existing systems for detecting abnormal stock market signals rely on synchronous polling, which limits real-time performance, scalability, and availability.
A system for detecting abnormal stock market movements based on asynchronous events, utilizing a low-frequency data synchronization module, data receiving adapter, abnormal indicator library, data storage module, and abnormal signal push module, with Redis for data transfer and asynchronous event model for high availability and scalability.
Ensures high availability, high performance, and easy expansion with low latency and multi-node deployment, supporting real-time detection and push of abnormal signals.
Smart Images

Figure CN119090639A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of financial data processing, in particular to the field of abnormal signal detection for real-time stock quotes, and specifically refers to a system for detecting abnormal signals for real-time stock quotes based on asynchronous events. Background Art
[0002] The stock market is a highly dynamic and complex system, in which rapid changes in stock prices and fluctuations in trading volume are often signals of market fluctuations. In order to capture these signals in a timely manner, investors and analysts need a system that can quickly respond to market changes. In the existing technology, synchronous polling from the market master station is often used to obtain the latest stock price and trading volume information to judge the abnormal signal, which has limitations in real-time, scalability, and high availability. Summary of the invention
[0003] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a system for detecting abnormal movements in real-time stock quotes based on asynchronous events, which has good real-time performance, high scalability and a wide range of applications.
[0004] In order to achieve the above purpose, the system of the present invention for detecting abnormal movement of real-time stock market based on asynchronous events is as follows:
[0005] The main features of the system for detecting abnormal movements in real-time stock quotes based on asynchronous events are as follows:
[0006] The low-frequency data synchronization module is used to connect to the main market station to synchronize and update the stock code table, basic stock information, and historical market data, and convert the format, and regularly update the low-frequency data to the data storage module;
[0007] The data receiving adapter module is used to connect to multiple market data sources, perform protocol analysis and data format conversion, convert real-time market data into a unified format, and issue market update notifications;
[0008] The abnormal indicator library module is used to define a variety of different abnormal signal judgment rules;
[0009] The data storage module is used to cache and persist intermediate data and result data for use by the low-frequency data synchronization module and the abnormal signal detection module;
[0010] The abnormal signal detection module is connected to the abnormal indicator library module and the data storage module, and is used to read the low-frequency data in a unified format, and perform indicator calculation and processing according to the abnormal indicator library rules and the real-time market data received through subscription;
[0011] The abnormal signal push module has an output end connected to the client, is used to provide data access services for the client, and receives the client's abnormal subscription request to establish a long connection, and pushes the abnormal signal data to the client in real time.
[0012] Preferably, the low-frequency data synchronization module is used to connect to the market master station, pull low-frequency data, synchronize the full stock code table, extract multiple data from the response data after parsing, convert them, and classify them by stock subcategories; synchronize historical market data, mainly refers to synchronizing the historical market K-line of the latest specified period, and update the formatted low-frequency data to the data storage module.
[0013] Preferably, the data receiving adaptation module comprises:
[0014] The data receiving unit is used to connect to multiple channels of real-time market data sources and distinguish market data sources from different markets and data centers;
[0015] A data parsing unit, connected to the data receiving unit, for parsing market source data and extracting multiple pieces of data from each frame of real-time data;
[0016] A structure conversion unit, connected to the data parsing unit, is used to convert the parsed market data and extract the type to which the data belongs;
[0017] A data deduplication unit, connected to the structure conversion unit, is used to deduplicate data and exclude market data that has not been updated but has been pushed;
[0018] The data updating unit is connected to the structure deduplication unit and is used for formatting data updating notification.
[0019] Preferably, the abnormal signal detection module comprises:
[0020] An initialization unit, connected to the data storage module, for loading a stock code set and its corresponding basic information and historical market data according to the configuration, and generating a variety of local cache indicators;
[0021] A data subscription unit, connected to the data storage module, for subscribing to data change notifications and obtaining updated market update data;
[0022] A data distribution unit, connected to the data subscription unit, for distributing the received market update data to the corresponding abnormal movement judgment unit;
[0023] The abnormality judgment unit is connected to the data distribution unit through an asynchronous event model, and is used to judge abnormality indicators of the received market data in combination with its own basic indicators and cached data, and update its own cache with real-time market data.
[0024] Preferably, the abnormal signal push module determines whether push is required according to the push logic, and the output end is connected to the client to provide data access service for the client, receive the abnormal subscription request from the client to establish a long connection, and push the abnormal signal data to the client in real time.
[0025] The system for detecting abnormal movement of real-time stock market based on asynchronous events of the present invention is adopted, and the speed priority and automatic fault switching of the data receiving link are realized by synchronizing data between receiving nodes, so as to ensure the high availability reception of the real-time market data receiving link; Redis is fully utilized as data transfer, and market update notifications are issued through the sub / pub mechanism, with low latency; abnormal movement detection fully utilizes the asynchronous event model, divides data processing into multiple relatively independent topics, supports lightweight deployment, and is convenient for expansion; abnormal movement signal push is ensured to be not pushed repeatedly through caching, supports multi-node deployment, and improves availability; in summary, the present invention has the obvious characteristics of high availability, high performance, and easy expansion. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 It is a structural schematic diagram of a system for detecting abnormal movements of real-time stock quotes based on asynchronous events according to the present invention.
[0027] Figure 2 The present invention is a schematic diagram of asynchronous event processing of a system for detecting abnormal movements in real-time stock quotes based on asynchronous events. DETAILED DESCRIPTION
[0028] In order to more clearly describe the technical content of the present invention, further description is given below in conjunction with specific embodiments.
[0029] like Figure 1 As shown, the system for detecting abnormal movements of real-time stock market information based on asynchronous events of the present invention includes a low-frequency data synchronization module for connecting to a market master station, performing synchronous update and format conversion processing of a stock code table, basic stock information, and historical market data, and regularly updating low-frequency data to a data storage module;
[0030] The data receiving adapter module is used to connect to multiple market data sources, perform protocol analysis and data format conversion, convert real-time market data into a unified format, and issue market update notifications;
[0031] The abnormal indicator library module is used to define a variety of different abnormal signal judgment rules;
[0032] The data storage module is used to cache and persist intermediate data and result data for use by the low-frequency data synchronization module and the abnormal signal detection module;
[0033] The abnormal signal detection module is connected to the abnormal indicator library module and the data storage module, and is used to read the low-frequency data in a unified format, and perform indicator calculation and processing according to the abnormal indicator library rules and the real-time market data received through subscription;
[0034] The abnormal signal push module has an output end connected to the client, is used to provide data access services for the client, and receives the client's abnormal subscription request to establish a long connection, and pushes the abnormal signal data to the client in real time.
[0035] As a preferred embodiment of the present invention, the low-frequency data synchronization module is used to connect to the market master station, pull low-frequency data, synchronize the full stock code table, extract multiple data from the response data after parsing, convert them, and classify them by stock subcategories; synchronize historical market data, mainly refers to synchronizing the historical market K-line of the latest specified period, and update the formatted low-frequency data to the data storage module.
[0036] As a preferred embodiment of the present invention, the data receiving adaptation module includes:
[0037] The data receiving unit is used to connect to multiple channels of real-time market data sources and distinguish market data sources from different markets and data centers;
[0038] A data parsing unit, connected to the data receiving unit, for parsing market source data and extracting multiple pieces of data from each frame of real-time data;
[0039] A structure conversion unit, connected to the data parsing unit, is used to convert the parsed market data and extract the type to which the data belongs;
[0040] A data deduplication unit, connected to the structure conversion unit, is used to deduplicate data and exclude market data that has not been updated but has been pushed;
[0041] The data updating unit is connected to the structure deduplication unit and is used for formatting data updating notification.
[0042] As a preferred embodiment of the present invention, the abnormal signal detection module includes:
[0043] An initialization unit, connected to the data storage module, for loading a stock code set and its corresponding basic information and historical market data according to the configuration, and generating a variety of local cache indicators;
[0044] A data subscription unit, connected to the data storage module, for subscribing to data change notifications and obtaining updated market update data;
[0045] A data distribution unit, connected to the data subscription unit, for distributing the received market update data to the corresponding abnormal movement judgment unit;
[0046] The abnormality judgment unit is connected to the data distribution unit through an asynchronous event model, and is used to judge abnormality indicators of the received market data in combination with its own basic indicators and cached data, and update its own cache with real-time market data.
[0047] As a preferred embodiment of the present invention, the abnormal signal push module determines whether push is required according to the push logic, and the output end is connected to the client to provide data access services for the client, and receives the client's abnormal subscription request to establish a long connection, and pushes the abnormal signal data to the client in real time.
[0048] In a specific embodiment of the present invention, Figure 2 As shown, as a preferred embodiment of the present invention, the abnormal signal detection module uses an asynchronous event model. When an event occurs, the event is added to the queue, and the background thread continuously consumes from the queue. After obtaining the event, the registered callback function is called in sequence according to the event handler list for notification.
[0049] The present invention comprises a plurality of modules: a low-frequency data synchronization module, a data storage module, a data receiving and adapting module, an abnormal indicator library module, an abnormal signal detection module, and an abnormal signal pushing module.
[0050] Low-frequency data synchronization module: used to connect to the market master station, synchronize the full stock code table, extract multiple data from the response data after parsing, convert them, and classify them by stock subcategories to form a subcategory corresponding code set, which is stored in the data storage module with the stock subcategory as the key; synchronize the basic stock information in batches according to the subcategory corresponding code set, including the security name, security code, circulating share capital, price limit, etc., and store them in the data storage module according to the stock level; synchronize the historical market data in batches according to the subcategory corresponding stock code set, mainly referring to the synchronization of the latest historical market K-line of the specified period, sort the K-line data by extraction date, and store them in the data storage module according to the stock level; this module is executed at a fixed period, and the latest data is always kept, and the historical data will be overwritten;
[0051] Data receiving adapter module: connects to multiple market sources of multiple exchanges, performs protocol analysis and data format conversion, converts market data into a unified format within the system, and then sends it to the redis cluster according to the market type. The data receiving adapter module publishes market update notifications in pub mode;
[0052] The specific processing steps of the data receiving adapter module are as follows:
[0053] This module connects to multiple real-time market sources, performs protocol adaptation and data conversion according to the access requirements of each data source, and converts upstream data into a unified format within the system.
[0054] For scenarios with multiple real-time market sources connected, this module receives multiple data simultaneously and extracts multiple data from each frame of real-time market data, converts the format and extracts the data type, and compares it with the previous tick data to remove duplicates. This module maintains a mark internally to determine whether the current update is the master node. The master node continuously updates the formatted data to the cache module. At the same time, this module maintains a real-time transaction volume of a configured target to the redis cache for switching judgment between the master and standby nodes. The standby node only checks the transaction volume in the redis cache based on the configured target, determines whether the current market speed is better than the master node, and triggers automatic switching of market updates when the speed is better. Only one copy of market data is written to the redis cache at the same time for the master and standby nodes, and publishes pub notifications.
[0055] Abnormal indicator library module: responsible for the specific rule calculation of abnormal indicators, including specific rules for abnormal types such as price surge / price plunge / near the upper limit / near the lower limit / upper limit / lower limit / rocket launch / high diving, etc.
[0056] Data storage module, used to cache and persist code tables, basic stock information, historical market K-lines, market snapshots, intermediate data, and result data;
[0057] The abnormal signal detection module is mainly used to read formatted low-frequency data, subscribe to real-time market data, and perform indicator calculation and processing based on the abnormal indicator library rules and the subscribed real-time market data;
[0058] The specific processing steps of the abnormal signal detection module are as follows:
[0059] This module loads the specified stock code set and its corresponding basic information and historical market data from redis according to the configured stock subcategory type, and generates local cache indicators such as yesterday's highest price, yesterday's lowest price, 5-day average price, 60-day highest price, 60-day lowest price, etc. based on the historical market data; subscribes to different topics from the data storage module and starts multiple abnormal movement detection coroutines.
[0060] After receiving the pub data, the module matches the stock code set it processes, selects the stock code list that needs to update the market TICK, and obtains the latest market TICK according to the stock code list and updates it to the asynchronous event model. After the asynchronous event model obtains the update event, it calls the registered callback function in sequence according to the event handler list for notification.
[0061] After receiving the event notification, the module's abnormality detection coroutine combines its own basic indicators and cached data to judge abnormal indicators, and updates its own cached real-time indicator data with real-time market data;
[0062] The abnormal signal push module has an output end connected to the client, is used to provide data access services for the client, and receives the client's abnormal subscription request to establish a long connection, and pushes the abnormal signal data to the client in real time.
[0063] The present invention comprises a plurality of steps:
[0064] 1. Synchronization of low-frequency data, including the synchronization update and format conversion processing of stock code table, stock basic information, and historical market data, and regular update of low-frequency data to the data storage module.
[0065] 2. Receiving and formatting multi-source real-time market data. The data receiving adapter module receives real-time data from multiple market data sources for analysis and publishes market update notifications.
[0066] 3. Detection of abnormal signals. The abnormal detection module reads the formatted low-frequency data, subscribes to the real-time market data, and calculates and processes the indicators according to the rules of the abnormal indicator library and the real-time market data received through subscription.
[0067] 4. Push of abnormal signal: real-time push of signals according to subscription rules.
[0068] The data storage module of the present invention is implemented based on Redis, which mainly transfers message data and performs subscription and push operations on different topics for real-time market data. Redis, as an independent and widely used memory data structure storage system, plays a pivotal role in the field of data processing. With its extremely fast read and write speed, rich data types and support for atomic operations, it has become the first choice for cache layer and data structure storage in many applications. Redis can not only effectively reduce the pressure on the database and increase the speed of data access, but also support complex data operations through its rich data structures (such as strings, lists, sets, hash tables, ordered sets, etc.), providing developers with great flexibility and convenience.
[0069] The subject mentioned in this manual, namely Topic, is not unique to the Redis platform. It is a common concept of message processing platforms (such as Kafka, Pulsar, RabbitMQ) and is used to maintain a class of message data.
[0070] The present invention provides a method for detecting abnormal signals based on an asynchronous event model, which has the following characteristics:
[0071] Low latency: the abnormality detection component is directly connected to the data receiving adapter module through a subscription mechanism, reducing the data circulation links;
[0072] Scalability: low-frequency data / real-time detection data / anomaly detection indicators are separated from each other, which is convenient for subsequent expansion;
[0073] High availability: the data receiving adapter module monitors the target, and the abnormality detection module automatically deduplicates the target signal, both of which support multi-node deployment.
[0074] The specific implementation scheme of this embodiment can refer to the relevant description in the above embodiment, which will not be repeated here.
[0075] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.
[0076] It should be noted that, in the description of the present invention, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" refers to at least two.
[0077] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0078] The system for detecting abnormal movement of real-time stock market based on asynchronous events of the present invention is adopted, and the speed priority and automatic fault switching of the data receiving link are realized by synchronizing data between receiving nodes, so as to ensure the high availability reception of the real-time market data receiving link; Redis is fully utilized as data transfer, and market update notifications are issued through the sub / pub mechanism, with low latency; abnormal movement detection fully utilizes the asynchronous event model, divides data processing into multiple relatively independent topics, supports lightweight deployment, and is convenient for expansion; abnormal movement signal push is ensured to be not pushed repeatedly through caching, supports multi-node deployment, and improves availability; in summary, the present invention has the obvious characteristics of high availability, high performance, and easy expansion.
[0079] In this specification, the present invention has been described with reference to specific embodiments thereof. However, it is apparent that various modifications and variations may be made without departing from the spirit and scope of the present invention. Therefore, the specification and drawings should be regarded as illustrative rather than restrictive.
Claims
1. A system for detecting abnormal changes in real-time stock quotes based on asynchronous events, characterized in that: The system comprises: The low-frequency data synchronization module is used to connect to the main market station to synchronize and update the stock code table, basic stock information, and historical market data, and convert the format, and regularly update the low-frequency data to the data storage module; The data receiving adapter module is used to connect to multiple market data sources, perform protocol analysis and data format conversion, convert real-time market data into a unified format, and issue market update notifications; The abnormal indicator library module is used to define a variety of different abnormal signal judgment rules; The data storage module is used to cache and persist intermediate data and result data for use by the low-frequency data synchronization module and the abnormal signal detection module; The abnormal signal detection module is connected to the abnormal indicator library module and the data storage module, and is used to read the low-frequency data in a unified format, and perform indicator calculation and processing according to the abnormal indicator library rules and the real-time market data received through subscription; The abnormal signal push module has an output end connected to the client, is used to provide data access services for the client, and receives the client's abnormal subscription request to establish a long connection, and pushes the abnormal signal data to the client in real time.
2. The system for detecting abnormal changes in real-time stock market information based on asynchronous events according to claim 1, characterized in that: The low-frequency data synchronization module is used to connect to the market master station, pull low-frequency data, synchronize the full stock code table, extract multiple data from the response data after parsing, convert them, and classify them by stock subcategories; synchronize historical market data, mainly refers to synchronizing the historical market K-line of the latest specified period, and update the formatted low-frequency data to the data storage module.
3. The system for detecting abnormal changes in real-time stock market information based on asynchronous events according to claim 1, characterized in that: The data receiving adapter module comprises: The data receiving unit is used to connect to multiple channels of real-time market data sources and distinguish market data sources from different markets and data centers; A data parsing unit, connected to the data receiving unit, for parsing market source data and extracting multiple pieces of data from each frame of real-time data; A structure conversion unit, connected to the data parsing unit, is used to convert the parsed market data and extract the type to which the data belongs; A data deduplication unit, connected to the structure conversion unit, is used to deduplicate data and exclude market data that has not been updated but has been pushed; The data updating unit is connected to the structure deduplication unit and is used for formatting data updating notification.
4. The system for detecting abnormal changes in real-time stock market conditions based on asynchronous events according to claim 1, characterized in that: The abnormal signal detection module comprises: An initialization unit, connected to the data storage module, for loading a stock code set and its corresponding basic information and historical market data according to the configuration, and generating a variety of local cache indicators; A data subscription unit, connected to the data storage module, for subscribing to data change notifications and obtaining updated market update data; A data distribution unit, connected to the data subscription unit, for distributing the received market update data to the corresponding abnormal movement judgment unit; The abnormality judgment unit is connected to the data distribution unit through an asynchronous event model, and is used to judge abnormality indicators of the received market data in combination with its own basic indicators and cached data, and update its own cache with real-time market data.
5. The system for detecting abnormal changes in real-time stock market information based on asynchronous events according to claim 1, characterized in that: The abnormal signal push module determines whether push is required according to the push logic. The output end is connected to the client to provide data access services for the client, receive the abnormal subscription request from the client to establish a long connection, and push the abnormal signal data to the client in real time.
