An intelligent partitioning system in a distributed scenario

Abstract

The present invention discloses an intelligent partitioning system in a distributed scenario. The bottom layer of the system includes computing nodes and is connected to a storage engine. The real-time incoming financial data flow is calculated and stored by the bottom layer of the system, and then the data processing information is transmitted to the cluster perception module. The cluster perception module is used to analyze the distribution of each key value and read-write pressure in the data; the obtained distribution is input into the intelligent prediction module, and the distribution of read-write requests for each key value in the data in the future is predicted in real time through an online learning algorithm; and the skewness of the hot data of the current cluster is evaluated. When the skewness exceeds the threshold, the intelligent partition migration module receives the prediction result of the prediction unit, calculates a new partition table, and performs a partition migration operation to achieve the purpose of balancing the future pressure of the cluster. The present invention is applicable to the business field of financial scenarios, and can significantly reduce the hotspot situation of the storage system and improve the response speed of the system.

Description

Technical Field

[0001] The present invention relates to the field of real-time processing of time series big data, and in particular to an intelligent partitioning system in a distributed scenario. Background Art

[0002] In the financial field, big data analysis has the characteristics of large data flow, low latency requirements, and strong data timeliness. Especially in certain specific scenarios, such as high-frequency quantitative trading, the distributed data analysis system needs to monitor hundreds of thousands of stocks and futures around the world. These stocks and futures will generate millions or even tens of millions of transaction flows per second. These data will be transmitted to the distributed analysis system, and after a series of calculations, the analysis results will be fed back. In the current high-frequency quantitative trading market, the opportunities for long and short positions are fleeting. If the data analysis system cannot stably feedback results at the sub-second level or even the millisecond level, it will cause immeasurable losses.

[0003] However, the popularity of different stocks varies greatly. The turnover rate of popular stocks may reach 7% or even exceed 10%, while the turnover rate of unpopular stocks may be less than 1%. If a large amount of popular stock data is routed to the same server in the cluster, the server will be overloaded and hot spots will appear, which will reduce the response speed of the server and affect the operation of actual business. Traditional partitioning methods usually use the experience of experts to screen out possible hot key values ​​and use hard coding to distribute them as evenly as possible to each server in the cluster, so as to reduce hot spots as much as possible. However, in actual financial scenarios, hot stocks may change over time. For example, with the emergence of new technologies, some technology stocks may suddenly turn from cold to hot. For example, some popular companies may suddenly have news of financial fraud, and their stocks will fall to the limit, resulting in a sharp drop in the amount of transaction flow data. These accidental events may cause the amount of stock transaction data to fluctuate dramatically. However, from the perspective of the entire macro stock market, these accidental events occur every day, which also causes the hot spot distribution of stock data to change every day. In this case, the static, hard-coded partitioning effect is extremely unstable. Summary of the invention

[0004] The purpose of the present invention is to address the deficiencies of the prior art and provide an intelligent partitioning system for distributed financial real-time analysis scenarios. The underlying layer can be connected to multiple storage engines, and based on the read and write request data of the past period of time, the future request distribution can be predicted, and the partitioning situation can be dynamically adjusted, thereby reducing the hot spots of the cluster and improving the response speed and operating efficiency of the cluster.

[0005] The object of the present invention is achieved through the following technical solutions: an intelligent partitioning system in a distributed scenario, the intelligent partitioning system comprising a cluster perception module, an intelligent prediction module and an intelligent partition migration module;

[0006] The bottom layer of the intelligent partitioning system includes computing nodes and is connected to the storage engine. The real-time financial data flow is calculated and stored by the bottom layer of the system, and then the data processing information is transmitted to the cluster perception module.

[0007] The cluster perception module analyzes the distribution of each key value and read / write pressure in the data; and inputs the obtained distribution into the intelligent prediction module;

[0008] The intelligent prediction module includes a prediction unit and an evaluation unit. The prediction unit predicts the distribution of read and write requests for each key value in the data in real time within a period of time in the future through an online learning algorithm. The evaluation unit uses the prediction result of the prediction unit as input to evaluate the skewness of the hot data of the current cluster.

[0009] When the skewness of hotspot data exceeds a threshold, the intelligent partition migration module receives the prediction result of the prediction unit, calculates a new partition table, and performs a partition migration operation to achieve the purpose of balancing the future pressure of the cluster.

[0010] Furthermore, the bottom layer of the system can be connected to a variety of storage engines, including redis, Aerospike, and Memcached.

[0011] Furthermore, the cluster perception module receives the processed data information, analyzes and counts it, and then sends it to the intelligent prediction module for intelligent prediction and evaluation, and the intelligent partition migration module adjusts the partitions of the cluster to balance the pressure of the distributed system. The data processing information includes the key value of the data and the time consumed to execute the data.

[0012] Furthermore, the cluster perception module classifies a processed piece of data information according to the key value and performs statistics respectively. For a key value, a time window of a certain length is maintained to count the total time consumed by the data belonging to the key value. The statistical results generated by the cluster perception module will be transmitted to the intelligent prediction module at a certain rate.

[0013] Furthermore, the intelligent prediction module receives the statistical results generated by the cluster perception module at a fixed rate, takes it as the data input of the prediction unit, performs online learning through the online learning algorithm, and predicts the total time consumed by the data of each key value in the future. Among them, the online learning algorithm is a pluggable component that runs as an independent process. The online learning algorithm includes an online RNN algorithm, an LSTM algorithm or a hidden Markov model. The intelligent prediction module will evaluate the prediction results generated by the algorithm through the evaluation unit, determine whether the pressure of the cluster is balanced in the future, and then determine whether the cluster needs to be rebalanced.

[0014] Furthermore, the intelligent partition migration module receives the prediction results generated by the prediction unit in the intelligent prediction module, calculates a new partition table, and uses the difference between the new and old partition tables to calculate the migration instructions required for the cluster rebalancing operation.

[0015] Furthermore, the intelligent partition migration module will comprehensively consider the pressure of the cluster in the future, design a parallel migration plan according to the migration instructions, and execute the migration. During the migration process, the intelligent partition migration module will monitor the pressure of the cluster in real time and limit the migration operation to ensure the real-time high availability of the cluster.

[0016] The beneficial effects of the present invention are as follows: the present invention designs a distributed intelligent partitioning system for financial scenarios, which can be connected to a variety of storage engines at the bottom layer, and can dynamically adjust the partitioning situation in real time to ensure the efficiency of the underlying computing nodes and storage; the present invention uses statistical methods and online learning algorithms to predict the pressure distribution of cluster partitions in the future, and disperses and migrates hot data in advance, ensuring that the underlying computing nodes and storage in the system also maintain low latency, high throughput and other characteristics of the cluster when the hot data distribution changes; in addition, the online learning algorithm is inserted into the online learning module as a pluggable component, which realizes the replaceability of the algorithm, can adapt to a variety of scenarios, and improves the robustness of the partitioning system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of the system structure of the present invention;

[0018] Figure 2 This is the structural diagram of the cluster perception module;

[0019] Figure 3 This is the structure diagram of the intelligent prediction module;

[0020] Figure 4 It is a cluster pressure distribution diagram in the embodiment;

[0021] Figure 5 A comparison diagram of the new and old partition schemes in the embodiment;

[0022] Figure 6 Schematic diagram of migration instructions and migration solutions in the embodiment. Specific implementation methods

[0023] The present invention is further described in detail below with reference to the accompanying drawings and specific examples.

[0024] The intelligent partitioning system is responsible for managing the partition status of the cluster. The underlying computing nodes can access a variety of different storage media, such as redis, Aerospike, Memcached, etc. The intelligent partitioning system is divided into three major modules, namely the cluster perception module, the intelligent prediction module and the intelligent partition migration module. Assume that the partition of the data received by the underlying computing node of the system belongs to the set Key = {key i |0≤i<n}, n is the size of the partition set, the format of the i-th financial data flow is data i =(k i , v i ), k i ∈Key. Where v i It is the data content of the financial data, including the time when the financial data occurred, the characteristics of the financial data, such as the stock code, and the detailed data of the financial data, such as the latest stock price and transaction status. i Indicates the partition to which the financial transaction data belongs, which is calculated using a hash algorithm based on the characteristics of the financial data.

[0025] 1) The cluster perception module is the entrance to the entire cluster and is responsible for monitoring the running status of the cluster. This module receives the input data of the cluster and records the time when the data enters the computing node (in_time). i At the same time, this module is also responsible for receiving the calculated results from the computing nodes and recording the time when the data calculation is completed out_time i This module uses this monitoring information to calculate the calculation time exec_time required for each data i , and use these data to statistically analyze the pressure distribution of the cluster in each time period and the pressure of each partition. The cluster perception module maintains a time window for each partition to count the data calculation time belonging to the partition in the past period of time, and inputs it into the message middleware at a certain frequency and sends it to the downstream module. Supported message middleware include Kafka, RocketMQ, etc.

[0026] 2) The intelligent prediction module is responsible for predicting the pressure distribution of the cluster in the future. The module receives the cluster monitoring data from the cluster perception module from Kafka, forwards the data to the pluggable online learning component of the prediction unit, performs real-time online learning, obtains the prediction data of the pressure of the cluster in the future, and the evaluation unit in the intelligent prediction module analyzes and evaluates the prediction results. The online learning component predicts the pressure distribution of each partition in the future. The online learning algorithm component in the module is external and pluggable. The online learning algorithm component runs in the cluster as a separate process and interacts with the intelligent prediction module through a predefined RESTful interface. The online algorithm component can use online RNN algorithm, LSTM algorithm, hidden Markov model and other algorithms, which can flexibly adapt to various financial scenarios and improve the accuracy of algorithm prediction. The intelligent prediction module regularly pushes the monitoring data of the cluster into the online learning algorithm component, and pulls the latest prediction situation for analysis and processing. When the intelligent prediction module predicts that the skewness of cluster pressure will exceed the threshold in the future, that is, when excessive hot data is concentrated in a few cluster servers, the intelligent partition migration module will be notified through Kafka to rebalance the cluster partitions.

[0027] 3) The intelligent partition migration module is responsible for reallocating the partitions in the cluster when the cluster is skewed, so as to achieve the purpose of rebalancing the cluster pressure. This module first designs a new cluster partition table based on the pressure distribution of cluster partitions in the future predicted by the intelligent prediction module. Next, the intelligent partition migration module compares the current cluster partition situation with the designed cluster partition table, calculates the migration instructions, and designs a parallel migration plan based on the correlation between the migration instructions. During the migration process, the module will detect the overall pressure of the cluster, and limit the migration process according to the situation to ensure that the availability of the cluster will not be affected during the partition migration process. After the partition migration is completed, the cluster rebalancing operation is completed, and the availability returns to the optimal state again.

[0028] The following describes the specific implementation method of the intelligent partitioning system in combination with actual use cases. In the field of high-frequency quantitative trading, the distributed data analysis system needs to continuously monitor the data of hundreds of thousands of stocks and futures transactions around the world, and receive millions of data streams per second. In this scenario, a stable, high-performance distributed intelligent partitioning system is required to ensure the efficiency and stability of cluster performance.

[0029] like Figure 1As shown in the figure, the computing node cluster in the intelligent partitioning system is composed of several redis servers, which are mainly responsible for computing, analyzing, and storing stock transaction details; and the intelligent partitioning system maintains the health of the cluster and reduces the impact of hot spots on the cluster. The intelligent partitioning system is divided into three modules, namely cluster perception module, intelligent prediction module, and intelligent partition migration module. Assume that the cluster has n computing nodes Cluster = {server i , 0≤i<n}. Each stock and futures will be mapped to a partition, and there are m partitions in the cluster. Key={key i |0≤i<m}, each partition will only be assigned to one redis server node. For example, in the current system, there are 5 computing nodes in the cluster, namely server0 to server4, and there are 20 partitions distributed on these computing nodes, namely key0 to key 19 The format of financial data flow is DataStream = {(key i , data i )|key i ∈Key}, where key i Represents the partition corresponding to this data flow, data i It is the data of this pipeline, which consists of two parts: data i =(index i , trade i ), where index i Is the stock code, trade i The data of stock transactions represented by this transaction. The partition key of this data is i It is based on the stock code index i , calculated using a hash algorithm. The redis server stores the transaction details of each stock, and uses these details to perform further analysis and calculation on the stock.

[0030] When a piece of financial data flows into the data (key i , data i ) flows into the cluster, it will be sensed by the cluster perception module and the time when the data enters the cluster will be recorded. i Similarly, the cluster awareness module will record the time out_time after the data is processed. i Using these two pieces of data, calculate the processing time exec_time of each piece of data in the cluster i =out_time i -in_time iAfter collecting the processing time of a large amount of financial data flow, the cluster perception module sorts these data according to the start time in_time i And the corresponding partition key i Perform statistics to count the total time of data processing in each partition in the past 60 seconds. This module pushes the statistical data time window to the message middleware Kafka at a frequency of every 60 seconds. The process is as follows: Figure 2 shown.

[0031] The intelligent prediction module listens to a specific Topic on Kafka. After receiving the cluster statistics pushed by the cluster awareness module, it forwards them to the independently running online learning algorithm component through the RESTful interface. The module is as follows: Figure 3 As shown. The data sent includes three fields. The time field indicates the time when the statistical data occurs, the key field indicates the partition corresponding to the statistical data, and time_windows indicates the statistical details of the partition in the time window during this period. The online learning component adopts the hidden Markov model algorithm, and uses statistical data to continuously perform incremental training. The intelligent prediction module requests prediction data from the online learning algorithm component once every 60 minutes. The online learning algorithm will return the predicted values ​​of the pressure of each partition in 20 partitions in the next 24 hours, as well as the pressure of the entire cluster every hour in the next 24 hours. The time span and time granularity are both parameters and can be adjusted. The predicted data includes three fields. The time field indicates the starting time of the predicted data, the key field indicates the partition represented by the predicted data, and predictions indicates the predicted value of the pressure of the partition during this period. After obtaining these predicted data, the intelligent prediction module will calculate the degree of skewness of the cluster, that is, the variance of the cluster, based on the situation of the server where the partition is located. As shown Figure 4 Under the distribution shown, when the skewness of the cluster is too large and exceeds the threshold, the migration of partitions within the cluster will be triggered to achieve cluster balance.

[0032] After receiving the cluster migration instruction from the upstream module, the intelligent partition migration module will use the current partition pressure prediction data to calculate the partition distribution after the cluster pressure is balanced, such as Figure 5 After determining the partitioning scheme, the intelligent partition migration module will arrange the migration. It will evaluate the correlation between different migration instructions and design a set of parallel and efficient migration schemes, such as Figure 6 Each time a migration instruction is executed to migrate a partition, the intelligent partition migration module will first obtain the pressure of the cluster in the next hour from the intelligent prediction module, and dynamically set the current limit during the partition migration process based on the pressure to ensure that the normal use of the cluster is not affected.

[0033] After the entire migration process is completed, the cluster will return to a stable state, and the cluster's computing and storage pressure will be evenly distributed on each computing node, allowing the performance of the entire system to return to the optimal value.

[0034] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modification and change made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. An intelligent partitioning system in a distributed scenario, characterized in that: The intelligent partitioning system includes a cluster perception module, an intelligent prediction module and an intelligent partition migration module; The bottom layer of the intelligent partitioning system includes computing nodes and is connected to the storage engine; the real-time financial data flow is calculated and stored by the bottom layer of the system, and then the data processing information is transmitted to the cluster perception module; The cluster perception module analyzes the distribution of each key value and read-write pressure in the data; the obtained distribution is input into the intelligent prediction module; the cluster perception module receives the processed data information, classifies it according to the key value, and performs statistics respectively. For a key value, a time window of a certain length is maintained to count the total time consumed by the data belonging to the key value. The statistical results generated by the cluster perception module will be transmitted to the intelligent prediction module at a certain rate for intelligent prediction and evaluation, and the intelligent partition migration module will adjust the partitions of the cluster to balance the pressure of the distributed system; The intelligent prediction module includes a prediction unit and an evaluation unit. The intelligent prediction module receives the statistical results generated by the cluster perception module at a fixed rate and uses them as data input for the prediction unit. The online learning algorithm is used to perform online learning to predict the distribution of read and write requests for each key value and the total time consumed by the data in the future. The online learning algorithm is a pluggable component that runs in the form of an independent process. The evaluation unit uses the prediction results of the prediction unit as input to evaluate the skewness of the hotspot data of the current cluster. The prediction results generated by the evaluation algorithm of the evaluation unit are used to determine whether the pressure of the cluster will be balanced in the future, and then determine whether the cluster needs to be rebalanced. When the skewness of hot data exceeds a threshold, the intelligent partition migration module receives the prediction result of the prediction unit, comprehensively considers the pressure of the cluster in the future, designs a parallel migration plan according to the migration instruction, calculates a new partition table, and executes the partition migration operation to achieve the purpose of balancing the future pressure of the cluster.

2. According to the distributed intelligent partitioning system of claim 1, it is characterized in that: The system bottom layer can be connected to a variety of storage engines, including redis, Aerospike, and Memcached.

3. According to the distributed intelligent partitioning system of claim 1, it is characterized in that: The online learning algorithm includes an online RNN algorithm, an LSTM algorithm or a hidden Markov model.

4. According to the distributed intelligent partitioning system of claim 1, it is characterized in that: The intelligent partition migration module receives the prediction results generated by the prediction unit in the intelligent prediction module, calculates a new partition table, and uses the difference between the new and old partition tables to calculate the migration instructions required for the cluster rebalancing operation.

5. According to the distributed intelligent partitioning system of claim 4, it is characterized in that: During the migration process, the intelligent partition migration module will monitor the cluster pressure in real time and limit the migration operation to ensure the real-time high availability of the cluster.

Images