Market quotation analysis method and system based on big data

Abstract

The invention relates to the field of market analysis, in particular to a market quotation analysis method and system based on big data, historical transaction data are acquired and cleaned in real time by deploying a data acquisition and preprocessing module in a distributed server cluster, and high-concurrency query and rapid retrieval are realized by using a distributed storage and retrieval unit. The time sequence prediction module adopts deep learning or a tree model on a computational node to train and deduce a price sequence, and outputs a trend prediction result; the abnormity and risk detection module identifies price abnormity and generates a risk detection value; and the optimization and feedback module synthesizes the results and dynamically adjusts parameters, outputs a comprehensive prediction index and returns the comprehensive prediction index to each module to update the model. The visualization and external interface unit generates a report and provides an API / SDK, so that a business system can fully utilize the system to realize accurate analysis and risk monitoring of market conditions. According to the invention, data processing efficiency and prediction accuracy are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of market analysis, and in particular to a market situation analysis method and system based on big data. Background Art

[0002] In today's financial field, market analysis has become an important supporting tool for securities, futures, foreign exchange and other asset transactions. With the development of informatization and Internet technology, computer models are widely used in the industry to study price trends and fluctuations, and assist investment institutions and individual investors in making decisions. Traditional market analysis is mostly based on a single data source (for example, only using historical transaction prices or trading volumes), and predicts future price trends through simple linear regression, moving average and other methods. This type of method has certain practicality in the early stage or when the data scale is small. It can quickly form data evidence for subjective judgments, and to a certain extent, it also helps traders make more reasonable buying and selling operations. In addition, some practitioners have also tried to combine Internet data for market sentiment analysis, but they are often limited to the complexity and high cost of data capture and cleaning, as well as the lack of multi-source heterogeneous data processing methods, making it difficult to obtain stable and efficient results in real trading.

[0003] As the market continues to expand, the increasingly large scale of transaction data poses a severe challenge to the traditional stand-alone analysis model. In order to cope with new demands such as high-frequency trading and programmed trading, many institutions have begun to deploy computing tasks on clusters or cloud platforms, manage massive data through distributed databases or file systems, and use parallel computing frameworks to conduct deeper statistical analysis and prediction of market conditions. Some teams have also introduced machine learning or deep learning algorithms in an attempt to make breakthroughs in prediction accuracy and analysis dimensions. The emergence of these emerging technologies has provided strong support for the upgrade of market analysis from manual experience to automation and intelligence, enabling financial institutions to better cope with the ever-changing market environment.

[0004] However, the market analysis system of the prior art has the following shortcomings: Lack of real-time processing capabilities for multi-dimensional big data: Traditional systems often run on a single machine or a simple distributed architecture. When faced with high-concurrency and high-frequency market data, they are prone to data processing delays or insufficient throughput, making it difficult to respond to market dynamics in real time.

[0005] Many existing methods only focus on one aspect, such as trend prediction based on historical prices or anomaly identification, without effectively linking the two in the same platform or process. This results in the prediction model losing accuracy once a sudden event occurs in the market, while simple anomaly detection cannot provide subsequent correction and prediction support.

[0006] After some technical solutions output the prediction results or risk detection results, they no longer continuously update the model itself, resulting in aging of model parameters or failure to adapt to new market conditions in a timely manner. The complexity and volatility of the financial market require the model to have adaptive characteristics, otherwise it will be difficult to maintain reliability and accuracy in long-term use. Summary of the invention

[0007] The purpose of the present invention is to provide a market situation analysis method and system based on big data to solve the technical problems raised in the above background technology.

[0008] Based on the above ideas, the present invention provides the following technical solutions: A market situation analysis system based on big data, comprising: Data acquisition and preprocessing module, distributed storage and retrieval unit, time series prediction module, anomaly and risk detection module, and optimization and feedback module; The data acquisition and preprocessing module is deployed in a distributed server cluster; it is used to obtain market-related data in real time, the data refers to historical transaction data, and the acquisition frequency can be customized; the data is deduplicated, standardized and normalized through distributed data cleaning, missing value processing and feature engineering to output the preprocessed price sequence; The distributed storage and retrieval unit is implemented based on the storage nodes and database management software provided by the distributed server cluster; the pre-processed price data is stored in a large-scale distributed storage medium, and high-concurrency query and fast retrieval are achieved through an index structure; The time series prediction module is deployed on the computing nodes of the distributed server cluster, and is used to execute the time series prediction algorithm based on the price dimension; obtain the price series from the distributed storage and retrieval unit, and train and infer it through the deep learning model or the tree-based regression model; and output the trend prediction result for the price series; The anomaly and risk detection module runs on the computing nodes of the distributed server cluster and is used to execute the price-based anomaly detection algorithm; obtain price data features from the distributed storage and retrieval unit, perform detection and identification, and output risk detection values; The optimization and feedback module runs on a distributed server cluster, and performs comprehensive calculation and dynamic weight adjustment on the price trend prediction results generated by the time series prediction module and the risk detection value generated by the anomaly and risk detection module; outputs the final market comprehensive index or prediction result, and feeds the result back to the above two modules for updating model parameters or feature selection; It also includes a visualization and external interface unit, which is deployed between a distributed server cluster and a user terminal or in a cloud environment, and generates a visualization report or trend chart using the prediction results and risk detection values; and includes an API / SDK interface for an external business system to call the analysis results; Distributed server clusters are used to efficiently acquire and preprocess massive amounts of historical transaction data, automatically remove noise and ensure data structuring. Distributed storage and retrieval units are used to achieve high-speed reading, writing and index retrieval of massive amounts of data, avoiding the capacity and retrieval efficiency bottlenecks brought about by traditional stand-alone storage. Both the time series prediction module and the anomaly and risk detection module can run in parallel on the cluster, significantly improving the analysis speed and the accuracy of prediction and detection. The optimization and feedback module links trend prediction with risk detection to form an adaptive closed-loop mechanism, which can schedule and update model parameters in a timely manner when the market fluctuates. The visualization and external interface unit provides business users and other systems with clear and intuitive result output and interface calling methods, making the entire system highly scalable and maintainable in large-scale, real-time market analysis scenarios.

[0009] Preferably, the price-based time series prediction algorithm executed by the time series prediction module comprises the following steps: S1. Obtain the cleaned and standardized price series from the distributed storage and retrieval unit; S2. Use sliding windows, time series differences or other preprocessing methods to extract features from price data; S3, generate a single-dimensional price time series feature vector for prediction; S4. On the distributed server cluster, LSTM is used to train the above single-dimensional time series features; S5. Capture long-term and short-term dependencies and nonlinear mapping relationships to improve the accuracy and stability of price trend prediction; S6. Outputting the next moment price prediction value generated by the training or inference process as the trend prediction result; S7. Perform error feedback based on real observation data and transmit it to the model for parameter update.

[0010] Through preprocessing and feature extraction steps, the noise and redundancy in the original market data can be effectively reduced, making the input data required by the model more concise and with higher information density; using LSTM for training can extract the potential rules of time series in long-term and short-term dependencies, and improve the grasp of price fluctuation trends; it realizes the closed loop of prediction and error feedback, so that the model parameters are continuously updated to adapt to the ever-changing market environment and meet the needs of accurate prediction in real-time or quasi-real-time scenarios.

[0011] Preferably, the price at the i-th moment is predicted by the following formula: h i =fLSTM(P i ,h i−1 ;θ1) y i =W o ×h i +b o Among them, P i represents the "price feature" input at the i-th moment, including the price at that moment or its derivative features, wherein the derivative features include the moving average and difference of the price; h i represents the hidden layer state of LSTM at the i-th moment, which is used to capture the short-term and long-term dependencies of price time series; fLSTM( ) is the recursive function of the LSTM network, which receives the input P i And the hidden state h at the previous moment i−1 , and the output is determined by the trainable parameters θ1 of the network.

[0012] θ1 represents all trainable parameters within the LSTM network, including the weights and biases related to the input gate, forget gate, output gate, etc. W o and b o are the weight and bias of the output layer respectively. i Used when mapping to scalar prediction values; y i Represents the prediction result of the future price at the i-th moment.

[0013] The single-dimensional price series is converted into a mapping relationship between hidden states and outputs, and the scalability and generalization ability of the model are guaranteed by means of trainable parameters in the formula; the network contains a gating mechanism that can perform more accurate feature selection and information retention on the price time series, and has strong adaptability to abnormal fluctuations; the formula is trained in parallel in a distributed environment, and no matter how large the data volume is, it can achieve relatively stable and low-latency prediction result output in high-concurrency scenarios.

[0014] Preferably, the price-based anomaly detection algorithm executed by the anomaly and risk detection module comprises the following steps: S1. If only single price data is used, the price value is directly used as the detection input; S2, using the isolation forest method based on random partitioning to score the abnormality of each record in the price series; S3. Output a preliminary abnormality score to identify possible sharp fluctuations or abnormal jumps in prices; S4. For some records with the highest or lowest abnormality scores, a secondary verification is performed in combination with the clustering algorithm; S5. Finally, the risk detection value is output and the result is sent to the optimization and feedback module for fusion.

[0015] Isolation forest is used for random partitioning to quickly locate relatively rare or extreme price change points, which has the characteristics of high efficiency and accuracy; secondary verification can further reduce the false alarm rate and missed alarm rate, ensuring that the final output risk detection value is more reliable; sending the risk detection value to the optimization and feedback module helps to realize the linkage between trend prediction and anomaly detection, and improve the overall system's perception and response capabilities to market fluctuations.

[0016] Preferably, the abnormality of the i-th price data is calculated by the following formula:

[0017]

[0018] Among them, p i Represents the i-th price record.

[0019] T k represents the kth random tree in the isolation forest, which is used to randomly split the price data and calculate the separation path; path_length(p i ,T k ) indicates that in the random tree T k In order to record the price p i The length of the path required to separate to the leaf node. The shorter the path, the more likely it is that the price record is inconsistent with the overall distribution and tends to be abnormal. avg_path_length(T k ) indicates that in the random tree T k The average separation path length of all samples in is used as the normalized reference value; S i Represents the cumulative relative path score of the i-th price record in K random trees. The larger the sum, the more abnormal the tendency is. ξ is a fixed threshold or offset used to adjust the center position of the Sigmoid function in order to fine-tune the anomaly score; e represents the natural Euler constant; α i Represents the final abnormality; when α i When it is greater than the preset warning value, it means that the price has changed significantly at that moment and requires further attention.

[0020] Based on the isolation forest, the Sigmoid function mapping is introduced to convert the cumulative relative path score S i The anomaly degree is converted into a range of 0-1, which is convenient for subsequent fusion or comparison with other indicators. The fixed threshold ξ is introduced into the Sigmoid function to provide additional flexibility, which can adapt to the different sensitivity of anomaly definitions under different market conditions. The entire calculation process can be executed in parallel in a distributed computing environment, and can maintain high computing efficiency and practicality even when the amount of price data is very large.

[0021] Preferably, the linkage optimization algorithm executed by the optimization and feedback module comprises the following steps: S1, receiving the trend prediction result price prediction value output by the time series prediction module, and the abnormality degree output by the abnormality and risk detection module; S2. Combine the two into a comprehensive indicator through statistical mapping to measure the performance of a single price in terms of both trend and potential risk; S3. Preset hyperparameters, wherein the hyperparameters include risk sensitivity coefficients, and dynamically adjust the trust in the price forecast value according to the magnitude of the abnormality; When the abnormality is significantly higher than the risk threshold, excessive trust in price forecasts is suppressed and risk warnings are highlighted; When the degree of abnormality is within the normal range, the trust in the price forecast value is enhanced to improve the accuracy of predictions of normal market price fluctuations.

[0022] S4. Compare the integrated comprehensive indicators with the real prices, or accept online feedback from external business systems; S5. Input the comparison results back into the time series prediction algorithm and anomaly detection algorithm to update their network weights or path division strategies, and continuously enhance the system's ability to adapt to changes in the price environment.

[0023] Taking the trend prediction results into consideration together with the degree of abnormality can enable the system to quickly adjust the prediction weight when prices fluctuate sharply or abnormal events occur, thereby enhancing sensitivity to sudden risks; by setting hyperparameters such as the risk sensitivity coefficient, it can better adapt to the risk preferences of different investors or institutions; a continuous self-correction mechanism is formed, which can continuously optimize the prediction and detection algorithms according to actual price trends or external feedback to keep them consistent with the latest market conditions; when executing this linkage algorithm in a distributed environment, it can quickly respond to market changes brought about by high-frequency trading data.

[0024] Preferably, the price prediction value of the trend prediction result and the abnormality of the risk detection value are integrated by the following formula to output the final comprehensive index: z i =y i ×(1−β×α i)+γ×α i y i Represents the predicted value of the price at the i-th moment by the time series prediction algorithm; α i Represents the risk detection value of the anomaly and risk detection algorithm for the i-th price record; β is an adjustable hyperparameter used to control the inhibitory weight of the risk detection value on the prediction result; the larger the β is, the stronger the inhibitory effect is when the abnormality is high; γ is an adjustable hyperparameter used to control the compensation or correction of the risk detection value to the prediction result; the larger γ is, the more significant the abnormality is. i The greater the contribution to the final result; z i It represents the comprehensive price index generated after fusion, which can reflect the correction results of sudden market fluctuations or abnormal risks. This indicator takes into account both trend prediction and abnormal risks, and is ultimately output to the visualization and external interface unit or business system.

[0025] Through a linear weighted approach, the prediction results and risk detection values ​​are organically integrated, and price predictions can be quickly corrected even in high-frequency or extreme volatility scenarios; the existence of the two hyperparameters β and γ allows users or systems to flexibly adjust the results of market analysis according to risk preferences, which can ensure robustness while retaining sensitive detection of sudden events; the fusion formula, combined with the feedback mechanism, can be quickly iterated and updated in real or simulated transactions, reducing the mismatch risk of the model in a complex financial environment, and further improving the accuracy and fault tolerance of market analysis.

[0026] A market situation analysis method based on big data, comprising: using the market situation analysis system based on big data.

[0027] Compared with the prior art, the present invention has the following beneficial effects: The time series prediction algorithm improves the learning depth of single-dimensional price data: by processing the historical price data with sliding windows, time series differences, etc., a feature vector with time series dependence is generated, and then LSTM (or other recurrent neural networks) is used to accurately capture the changing patterns of prices in the short and long term. Compared with traditional linear models (such as ARIMA), it can significantly reduce the prediction error under nonlinear and high-volatility market conditions, and better reflect the sudden jumps and trend turning points in the price curve. Since the algorithm is deployed on a distributed server cluster, it can parallelize the processing and model training of massive historical transaction data, effectively improving the processing capabilities of large-scale and high-frequency market conditions. When the market sees a large-scale influx of data or sudden changes in the market, the system can also maintain high throughput and low latency with the help of high-performance computing nodes to ensure the timeliness and accuracy of the prediction. After the prediction is completed, the internal parameters of the model (such as network weights, gating thresholds, etc.) can be continuously updated through the error feedback mechanism formed by comparing with the real market price. Such adaptive adjustments enable the prediction algorithm to dynamically respond to the phased changes or sudden shocks of market conditions, reduce model obsolescence and offset, and significantly improve the tracking ability and practical value of subsequent price trends.

[0028] Abnormal price fluctuations can be efficiently identified through anomaly and risk detection algorithms, and high-risk situations can be warned in a timely manner: through random partitioning methods such as isolation forests, the price data is scored for abnormality, and clustering algorithms are used for secondary verification, which can quickly locate and mark extreme trends, flash crashes or abnormal trading behaviors. When prices show abnormal jumps or extraordinary fluctuations, high-risk warnings can be given in a short time, providing an important basis for subsequent algorithms or regulatory actions. The Sigmoid function and parameter ξ are introduced to allow the center position of the abnormality score to be adjusted to match different trading scenarios or risk control strategies. For example, in a market with more intense fluctuations, ξ can be set slightly higher to reduce false positives; while in a steady-state environment, ξ can be lowered to enhance the sensitivity to a few abnormal market movements. This flexibility improves the compatibility and configurability of the system in multiple business scenarios. The initial scoring of the isolation forest can quickly process large quantities of market data, and then the high-risk samples can be finely tested for secondary detection through density-based clustering methods (such as DBSCAN), which can significantly suppress both false positives and missed negatives. Such a multi-level detection structure further improves accuracy while ensuring detection efficiency, truly realizing timely and accurate capture of abnormal market conditions.

[0029] The optimization and feedback algorithm comprehensively considers trends and anomalies to form a more robust market index: by mathematically mapping or weighted fusion of the output price trend forecast value and the risk detection value, the overall market trend and local extreme changes can be taken into account; especially when the price deviates significantly or the risk signal increases significantly, the original forecast results can be dynamically corrected to provide a more realistic comprehensive market index. By setting a series of adjustable hyperparameters β and γ, the trust in the forecast results or the magnification factor of the risk value can be automatically adjusted according to market changes. This dynamic mechanism enables the system to flexibly change the algorithm tendency according to the external environment, user risk preference or market volatility, and maintain a moderate response to different levels of risk scenarios. Comparing the fused results with the actual price again or receiving external business feedback and passing them back to the first two algorithms can enable the trend forecast model and the anomaly detection model to benefit from the latest market data and identification results at the same time. In this cyclic update process, the mismatch or error of the two algorithms is continuously reduced, helping the system maintain a high-precision perception of the current market environment. At the same time, the loop design can also be quickly iterated in real trading or simulated testing environments to ensure that the system remains robust and efficient in the face of rapidly changing market conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The present invention is a schematic diagram of the module relationship of a market situation analysis method and system based on big data. DETAILED DESCRIPTION

[0031] A market situation analysis system based on big data, comprising: Data acquisition and preprocessing module, distributed storage and retrieval unit, time series prediction module, anomaly and risk detection module, and optimization and feedback module; The data acquisition and preprocessing module is deployed in a distributed server cluster; it is used to obtain market-related data in real time, the data refers to historical transaction data, and the acquisition frequency can be customized; the data is deduplicated, standardized and normalized through distributed data cleaning, missing value processing and feature engineering to output the preprocessed price sequence; The distributed storage and retrieval unit is implemented based on the storage nodes and database management software provided by the distributed server cluster; the pre-processed price data is stored in a large-scale distributed storage medium, and high-concurrency query and fast retrieval are achieved through an index structure; The time series prediction module is deployed on the computing nodes of the distributed server cluster, and is used to execute the time series prediction algorithm based on the price dimension; obtain the price series from the distributed storage and retrieval unit, and train and infer it through the deep learning model or the tree-based regression model; and output the trend prediction result for the price series; The anomaly and risk detection module runs on the computing nodes of the distributed server cluster and is used to execute the price-based anomaly detection algorithm; obtain price data features from the distributed storage and retrieval unit, perform detection and identification, and output risk detection values; The optimization and feedback module runs on a distributed server cluster, and performs comprehensive calculation and dynamic weight adjustment on the price trend prediction results generated by the time series prediction module and the risk detection value generated by the anomaly and risk detection module; outputs the final market comprehensive index or prediction result, and feeds the result back to the above two modules for updating model parameters or feature selection; It also includes a visualization and external interface unit, which is deployed between a distributed server cluster and a user terminal or in a cloud environment, and generates a visualization report or trend chart using the prediction results and risk detection values; and includes an API / SDK interface for an external business system to call the analysis results; Distributed server clusters are used to efficiently acquire and preprocess massive amounts of historical transaction data, automatically remove noise and ensure data structuring. Distributed storage and retrieval units are used to achieve high-speed reading, writing and index retrieval of massive amounts of data, avoiding the capacity and retrieval efficiency bottlenecks brought about by traditional stand-alone storage. Both the time series prediction module and the anomaly and risk detection module can run in parallel on the cluster, significantly improving the analysis speed and the accuracy of prediction and detection. The optimization and feedback module links trend prediction with risk detection to form an adaptive closed-loop mechanism, which can schedule and update model parameters in a timely manner when the market fluctuates. The visualization and external interface unit provides business users and other systems with clear and intuitive result output and interface calling methods, making the entire system highly scalable and maintainable in large-scale, real-time market analysis scenarios.

[0032] Configure multiple high-performance servers, each with a multi-core CPU, sufficient memory, and several dedicated acceleration hardware such as GPU / TPU; use distributed file systems (such as Hadoop HDFS) or distributed databases (such as HBase, Cassandra) to provide high-concurrency reading, writing, and index query of massive transaction data; connect computing nodes and storage nodes through Gigabit or 10 Gigabit high-speed network switches to ensure bandwidth and latency requirements for large-scale data throughput within the cluster.

[0033] The data collection and preprocessing module is deployed on several computing nodes of the distributed server cluster, and cooperates with the distributed data processing framework (such as Spark or MapReduce) to parallelize the cleaning, deduplication, missing value filling and feature engineering (such as standardization and normalization) of historical price data obtained from exchanges or third-party data sources, and finally outputs a well-structured "price series" and stores it in the distributed file system.

[0034] The distributed storage and retrieval unit is mainly implemented on the storage node. Using index technology (based on column storage index, B+ tree, LSM tree, etc.), it supports high-concurrency query of "price series" data. The time series prediction module and the anomaly and risk detection module can quickly read the target data fragments in batches or streaming when training and inference are required.

[0035] Specifically, the price-based time series prediction algorithm executed by the time series prediction module includes the following steps: S1. Obtain the cleaned and standardized price series from the distributed storage and retrieval unit; S2. Use sliding windows, time series differences or other preprocessing methods to extract features from price data; S3, generate a single-dimensional price time series feature vector for prediction; S4. On the distributed server cluster, LSTM is used to train the above single-dimensional time series features; S5. Capture long-term and short-term dependencies and nonlinear mapping relationships to improve the accuracy and stability of price trend prediction; S6. Outputting the next moment price prediction value generated by the training or inference process as the trend prediction result; S7. Perform error feedback based on real observation data and transmit it to the model for parameter update.

[0036] Through preprocessing and feature extraction steps, the noise and redundancy in the original market data can be effectively reduced, making the input data required by the model more concise and with higher information density; using LSTM for training can extract the potential rules of time series in long-term and short-term dependencies, and improve the grasp of price fluctuation trends; it realizes the closed loop of prediction and error feedback, so that the model parameters are continuously updated to adapt to the ever-changing market environment and meet the needs of accurate prediction in real-time or quasi-real-time scenarios.

[0037] Specifically, the price at the i-th moment is predicted by the following formula: h i =fLSTM(P i ,h i−1 ;θ1) y i =W o ×h i +b o Among them, P i represents the "price feature" input at the i-th moment, including the price at that moment or its derivative features, wherein the derivative features include the moving average and difference of the price; h irepresents the hidden layer state of LSTM at the i-th moment, which is used to capture the short-term and long-term dependencies of price time series; fLSTM( ) is the recursive function of the LSTM network, which receives the input P i And the hidden state h at the previous moment i−1 , and the output is determined by the trainable parameters θ1 of the network.

[0038] θ1 represents all trainable parameters within the LSTM network, including the weights and biases related to the input gate, forget gate, output gate, etc. W o and b o are the weight and bias of the output layer respectively. i Used when mapping to scalar prediction values; y i Represents the prediction result of the future price at the i-th moment.

[0039] The single-dimensional price series is converted into a mapping relationship between hidden states and outputs, and the scalability and generalization ability of the model are guaranteed by means of trainable parameters in the formula; the network contains a gating mechanism that can perform more accurate feature selection and information retention on the price time series, and has strong adaptability to abnormal fluctuations; the formula is trained in parallel in a distributed environment, and no matter how large the data volume is, it can achieve relatively stable and low-latency prediction result output in high-concurrency scenarios.

[0040] Specific implementation steps: Read the pre-processed “price sequence” {P1,P2,⋯ ,P N}, where each P i The price information corresponding to a moment; On distributed computing nodes, sliding windows or time-series differencing (such as P i −P i−1 ) and other methods to construct the training data set.

[0041] For each time i, take the price of the previous w times as the input feature vector: X i =(P i−w ,P i−w+1 ,…,P i−1 ) Optionally add moving average features (such as MA(P i ,n), differential features (such as P i −P i−1 ) etc., to enhance the ability to describe short-term price fluctuations.

[0042] On the GPU-accelerated nodes or multi-core CPU nodes of the cluster, use LSTM (Long Short-Term Memory Network) to train the above feature vectors: initialize the network weights and biases; Read training samples in batches (the batch size can be set to B) and send them to the LSTM network in parallel for forward propagation; Calculate the loss between the predicted output and the true price (you can use mean squared error or mean absolute error); Use the back-propagation algorithm to update the LSTM parameters; Iterate until the accuracy is met or the set number of rounds is reached.

[0043] After the training is completed, the input feature X at the i-th moment can be used in the inference phase. i Perform forward propagation and output the predicted value y i as an estimate of the subsequent price at that moment.

[0044] Real-time monitoring of new price data in the market, and i Compare and get the error; This error is used to trigger local retraining (such as micro-batch updates) in a distributed environment to ensure that the model adapts to the market environment in a timely manner.

[0045] The working principle of LSTM in time series is different from that of traditional RNN. LSTM adds a gating mechanism (such as input gate, forget gate, output gate) when updating the hidden state, which can effectively remember or forget historical information. Therefore, when learning the sequence, it can maintain sensitivity to distant past information and avoid the problem of gradient vanishing or gradient exploding.

[0046] The output (hidden state) at each moment i will affect the input at the next moment i+1. This cyclic recursive structure is very suitable for processing time series or sequence data.

[0047] In this formula, y i It can be the price prediction value at time i+1, or it can be converted into y by sliding window or other strategies. i It is regarded as a local target in multi-step prediction and used for continuous prediction at subsequent moments; The hidden state h of the LSTM network i As a high-dimensional representation. Through linear mapping (i.e. W o ×h i +b o ), which can transform this high-dimensional representation into the scalar or vector prediction value (such as price, transaction volume, etc.) required by the actual business.

[0048] W o and b oAs a trainable parameter, it can be learned according to the true distribution of the target value, so that the model gradually approaches the optimal prediction effect in long-term training.

[0049] P i Represents the input features at the i-th moment, usually the price data at that moment and its nearby moments (may also include simple derivative features, such as price difference, moving average, etc.). Its function is to provide the most basic time step information for LSTM, so that the network can learn and predict based on the historical price change trend.

[0050] h i represents the hidden state of LSTM at the i-th moment. It is the intrinsic representation of the price or sequence state at that moment extracted by the network, which contains the key information transmitted from the previous moment and the integration of the input information at the current moment. Compared with the hidden state of ordinary RNN, h in LSTM i Combining gating and cell state mechanisms can better prevent gradient decay or explosion in long sequences.

[0051] h i−1 Represents the hidden state of the previous moment (i-1), which is the core of the LSTM recursive structure. Its function is to bring the past memory to the current moment and combine it with the current input P i Working together, the new hidden state h is generated through the fLSTM() function i .

[0052] θ1 represents the set of all trainable parameters in the LSTM network, including all weights and biases such as the input gate, forget gate, and output gate. During the training process, these parameters will be continuously adjusted through the backpropagation algorithm (especially BPTT, Backpropagation Through Time) to minimize the prediction error and learn the long-term change rules of the price series.

[0053] W o and b o b are the weight matrix and bias vector of the output layer, respectively, used to transform the hidden state h i Convert to predicted value.

[0054] Its role is to achieve a linear mapping from high-dimensional hidden representation to target price. If the target to be predicted is multi-dimensional (such as multiple indicators), W can be adjusted accordingly. o shape.

[0055] y i Represents the predicted value of the price at the ith moment (or the next moment), which is the direct output of the entire time series prediction algorithm. In specific applications, y iIt may be passed to other modules (such as anomaly detection modules or compared with true values ​​for error feedback), or directly used to assist business needs such as transaction decision-making and risk assessment.

[0056] Specifically, the price-based anomaly detection algorithm executed by the anomaly and risk detection module includes the following steps: S1. If only single price data is used, the price value is directly used as the detection input; S2, using the isolation forest method based on random partitioning to score the abnormality of each record in the price series; S3. Output a preliminary abnormality score to identify possible sharp fluctuations or abnormal jumps in prices; S4. For some records with the highest or lowest abnormality scores, a secondary verification is performed in combination with the clustering algorithm; S5. Finally, the risk detection value is output and the result is sent to the optimization and feedback module for fusion.

[0057] Isolation forest is used for random partitioning to quickly locate relatively rare or extreme price change points, which has the characteristics of high efficiency and accuracy; secondary verification can further reduce the false alarm rate and missed alarm rate, ensuring that the final output risk detection value is more reliable; sending the risk detection value to the optimization and feedback module helps to realize the linkage between trend prediction and anomaly detection, and improve the overall system's perception and response capabilities to market fluctuations.

[0058] Specifically, the abnormality of the i-th price data is calculated by the following formula:

[0059]

[0060] Among them, p i Represents the i-th price record.

[0061] T k represents the kth random tree in the isolation forest, which is used to randomly split the price data and calculate the separation path; path_length(p i ,T k ) indicates that in the random tree T k In order to record the price p i The length of the path required to separate to the leaf node. The shorter the path, the more likely it is that the price record is inconsistent with the overall distribution and tends to be abnormal. avg_path_length(T k ) indicates that in the random tree T k The average separation path length of all samples in is used as the normalized reference value; S i Represents the cumulative relative path score of the i-th price record in K random trees. The larger the sum, the more abnormal the tendency is. ξ is a fixed threshold or offset used to adjust the center position of the Sigmoid function in order to fine-tune the anomaly score; e represents the natural Euler constant; α i Represents the final abnormality; when α i When it is greater than the preset warning value, it means that the price has changed significantly at that moment and requires further attention.

[0062] Based on the isolation forest, the Sigmoid function mapping is introduced to convert the cumulative relative path score S i The anomaly degree is converted into a range of 0-1, which is convenient for subsequent fusion or comparison with other indicators. The fixed threshold ξ is introduced into the Sigmoid function to provide additional flexibility, which can adapt to the different sensitivity of anomaly definitions under different market conditions. The entire calculation process can be executed in parallel in a distributed computing environment, and can maintain high computing efficiency and practicality even when the amount of price data is very large.

[0063] The implementation steps of the anomaly and risk detection algorithm include: Also on the distributed server cluster, read the normalized or standardized price series {p1,p2,⋯,p N}.

[0064] If only a single price dimension is considered, then p i As input; when there are multi-dimensional auxiliary data (such as trading volume), a small number of feature combinations can be added, but this embodiment only illustrates a single price dimension.

[0065] Using the Isolation Forest algorithm: randomly sample and construct several random trees T in a distributed environment k ; For each price record p i , in each tree, calculate the path length path_length(p i ,T k ); S is obtained by summing the relative lengths of the separation paths of all random trees. i .

[0066] S i Gmoid mapping and anomaly calculation The anomaly α is obtained by the formula i ; and if necessary, combine clustering (such as DBSCAN) to further confirm the high abnormality samples.

[0067] αi Combined with information such as whether the warning threshold is exceeded, the final "risk detection value" is given and synchronously transmitted to the optimization and feedback module.

[0068] The setting of ξ determines the degree of translation of the abnormality curve, allowing risk control strategies in different market environments or with different tolerances to customize abnormality judgment criteria, which is a flexible and creative design; In a highly volatile market, ξ can be appropriately increased to avoid frequent false alarms; in a steady-state market, ξ can be lowered to capture milder anomalies.

[0069] When α i When the warning threshold (such as 0.8) is exceeded, the system will mark the price at that moment as a high-risk point and trigger subsequent risk control or reminder business logic; You can also use α i The changing trends at multiple consecutive moments are used to identify abnormal increases and flash crashes, and provide a basis for decision-making for transaction supervision or automated trading strategies.

[0070] Build an isolated forest on distributed GPU or CPU nodes: Each random tree can be completed independently on one or more computing nodes and the results can be merged to meet the real-time processing needs of large-scale data; random sampling and parallel path calculation can make full use of the multi-threading and multi-tasking capabilities of distributed clusters, so that the algorithm can still run stably in ultra-large data volume scenarios.

[0071] Specifically, the linkage optimization algorithm executed by the optimization and feedback module includes the following steps: S1, receiving the trend prediction result price prediction value output by the time series prediction module, and the abnormality degree output by the abnormality and risk detection module; S2. Combine the two into a comprehensive indicator through statistical mapping to measure the performance of a single price in terms of both trend and potential risk; S3. Preset hyperparameters, wherein the hyperparameters include risk sensitivity coefficients, and dynamically adjust the trust in the price forecast value according to the magnitude of the abnormality; When the abnormality is significantly higher than the risk threshold, excessive trust in price forecasts is suppressed and risk warnings are highlighted; When the degree of abnormality is within the normal range, the trust in the price forecast value is enhanced to improve the accuracy of predictions of normal market price fluctuations.

[0072] S4. Compare the integrated comprehensive indicators with the real prices, or accept online feedback from external business systems; S5. Input the comparison results back into the time series prediction algorithm and anomaly detection algorithm to update their network weights or path division strategies, and continuously enhance the system's ability to adapt to changes in the price environment.

[0073] Taking the trend prediction results into consideration together with the degree of abnormality can enable the system to quickly adjust the prediction weight when prices fluctuate sharply or abnormal events occur, thereby enhancing sensitivity to sudden risks; by setting hyperparameters such as the risk sensitivity coefficient, it can better adapt to the risk preferences of different investors or institutions; a continuous self-correction mechanism is formed, which can continuously optimize the prediction and detection algorithms according to actual price trends or external feedback to keep them consistent with the latest market conditions; when executing this linkage algorithm in a distributed environment, it can quickly respond to market changes brought about by high-frequency trading data.

[0074] Specifically, the price prediction value of the trend prediction result and the abnormality of the risk detection value are integrated through the following formula to output the final comprehensive index: z i =y i ×(1−β×α i )+γ×α i y i Represents the predicted value of the price at the i-th moment by the time series prediction algorithm; α i Represents the risk detection value of the anomaly and risk detection algorithm for the i-th price record; β is an adjustable hyperparameter used to control the inhibitory weight of the risk detection value on the prediction result; the larger the β is, the stronger the inhibitory effect is when the abnormality is high; γ is an adjustable hyperparameter used to control the compensation or correction of the risk detection value to the prediction result; the larger γ is, the more significant the abnormality is. i The greater the contribution to the final result; z i It represents the comprehensive price index generated after fusion, which can reflect the correction results of sudden market fluctuations or abnormal risks. This indicator takes into account both trend prediction and abnormal risks, and is ultimately output to the visualization and external interface unit or business system.

[0075] Through a linear weighted approach, the prediction results and risk detection values ​​are organically integrated, and price predictions can be quickly corrected even in high-frequency or extreme volatility scenarios; the existence of the two hyperparameters β and γ allows users or systems to flexibly adjust the results of market analysis according to risk preferences, which can ensure robustness while retaining sensitive detection of sudden events; the fusion formula, combined with the feedback mechanism, can be quickly iterated and updated in real or simulated transactions, reducing the mismatch risk of the model in a complex financial environment, and further improving the accuracy and fault tolerance of market analysis.

[0076] The implementation steps of the optimization and feedback algorithm are: Get the predicted price y from the time series forecasting algorithm i ; Obtain the anomaly degree α from the anomaly and risk detection algorithmi .

[0077] On the distributed computing nodes, the two are merged according to the set fusion formula or statistical model, and the fusion weight can be adjusted dynamically.

[0078] Set the hyperparameters β and γ to adjust y according to different risk conditions i Make corrections; when α i If it is too high, increase α i The proportion in the fusion result, weakening the effect of y i trust; when α i When in the safe zone, more reliance on y i to obtain a more accurate trend estimate.

[0079] Get z i And output to the outside; at the same time, z i Compare it with the real price again and send the error back to the aforementioned algorithm to update its parameters or detection threshold.

[0080] Using new market data or external business system feedback, the LSTM network and isolation forest model are incrementally trained or periodically retrained simultaneously to form an adaptive closed loop.

[0081] The parameters β and γ give the system the ability to adjust to different risk tolerances: if β is set high, once α i The increase will greatly inhibit y i The influence of γ is suitable for risk-averse strategies. If γ is set appropriately, the price forecast value can be slightly revised, or strongly revised in extreme cases. This is an improvement on the traditional single prediction or single risk monitoring method, avoiding the fragmentation of information with only trend values ​​or only abnormal scores, and significantly enhancing the overall control over the market.

[0082] z i It can be regarded as a risk-corrected price forecast, which directly affects the buying and selling timing judgment of the automated trading system; In the risk management scenario, z i The price is continuously compared with the actual price. Once the error exceeds a certain range, more safety strategies will be called upon (such as large-scale order cancellation, reduction of leverage, etc.) to provide more robust risk control protection.

[0083] Calculation example: Assumption: y i =100;α i =0.3; β=1.5; γ=2.0.

[0084] Then: z i=100×(1−1.5×0.3)+2.0×0.3=55.6 In this scenario, an abnormality of 0.3 leads to a significant suppression of the original forecast value (from 100 to 55.6) because β is high; and γ of 2.0 has a certain compensatory effect on the abnormality. i Climbing, then z i It may be further significantly lowered to help the system respond to risks in a timely manner.

[0085] A market situation analysis method based on big data, comprising: using the market situation analysis system based on big data.

Claims

1. A market analysis system based on big data, characterized in that: include: Data acquisition and preprocessing module, distributed storage and retrieval unit, time series prediction module, anomaly and risk detection module, and optimization and feedback module; The data acquisition and preprocessing module is deployed in a distributed server cluster; it is used to obtain market-related data in real time, the data refers to historical transaction data, and the acquisition frequency can be customized; the data is deduplicated, standardized and normalized through distributed data cleaning, missing value processing and feature engineering to output the preprocessed price sequence; The distributed storage and retrieval unit is implemented based on the storage nodes and database management software provided by the distributed server cluster; the pre-processed price data is stored in a large-scale distributed storage medium, and high-concurrency query and fast retrieval are achieved through an index structure; The time series prediction module is deployed on the computing nodes of the distributed server cluster, and is used to execute the time series prediction algorithm based on the price dimension; obtain the price series from the distributed storage and retrieval unit, and train and infer it through the deep learning model or the tree-based regression model; and output the trend prediction result for the price series; The anomaly and risk detection module runs on the computing nodes of the distributed server cluster and is used to execute the price-based anomaly detection algorithm; obtain price data features from the distributed storage and retrieval unit, perform detection and identification, and output risk detection values; The optimization and feedback module runs on a distributed server cluster, and performs comprehensive calculation and dynamic weight adjustment on the price trend prediction results generated by the time series prediction module and the risk detection value generated by the anomaly and risk detection module; outputs the final market comprehensive index or prediction result, and feeds the result back to the above two modules for updating model parameters or feature selection; It also includes a visualization and external interface unit, which is deployed between the distributed server cluster and the user terminal or in a cloud environment, and generates a visualization report or trend chart using the prediction results and risk detection values; Includes API / SDK interface for external business system to call analysis results.

2. A market analysis system based on big data according to claim 1, characterized in that: The price-based time series prediction algorithm performed by the time series prediction module includes the following steps: S1. Obtain the cleaned and standardized price series from the distributed storage and retrieval unit; S2. Use sliding windows, time series differences or other preprocessing methods to extract features from price data; S3, generate a single-dimensional price time series feature vector for prediction; S4. On the distributed server cluster, LSTM is used to train the above single-dimensional time series features; S5. Capture long-term and short-term dependencies and nonlinear mapping relationships to improve the accuracy and stability of price trend prediction; S6. Outputting the next moment price prediction value generated by the training or inference process as the trend prediction result; S7. Perform error feedback based on real observation data and transmit it to the model for parameter update.

3. The market analysis system based on big data according to claim 2 is characterized in that: The price at the i-th moment is predicted by the following formula: h i =fLSTM(P i ,h i−1 ;θ1); y i =W o ×h i +b o ; Among them, P i Represents the "price feature" input at the ith moment, including the price at that moment or its derivative features, wherein the derivative features include the moving average and difference of the price; h i represents the hidden layer state of LSTM at the i-th moment, which is used to capture the short-term and long-term dependencies of price time series; fLSTM( ) is the recursive function of the LSTM network, which receives the input P i And the hidden state h at the previous moment i−1 , and the output is determined by the trainable parameter θ1 of the network; θ1 represents all trainable parameters within the LSTM network, including the weights and biases related to the input gate, forget gate, output gate, etc. W o and b o are the weight and bias of the output layer respectively. i Used when mapping to scalar prediction values; y i Represents the prediction result of the future price at the i-th moment.

4. The market analysis system based on big data according to claim 3 is characterized in that: The price-based anomaly detection algorithm implemented by the anomaly and risk detection module includes the following steps: S1. If only single price data is used, the price value is directly used as the detection input; S2, using the isolation forest method based on random partitioning to score the abnormality of each record in the price series; S3. Output a preliminary abnormality score to identify possible sharp fluctuations or abnormal jumps in prices; S4. For some records with the highest or lowest abnormality scores, a secondary verification is performed in combination with the clustering algorithm; S5. Finally, the risk detection value is output and the result is sent to the optimization and feedback module for fusion.

5. The market analysis system based on big data according to claim 4 is characterized in that: The abnormality of the i-th price data is calculated using the following formula: ； ； Among them, p i represents the i-th price record; T k represents the kth random tree in the isolation forest, which is used to randomly split the price data and calculate the separation path; path_length(p i ,T k ) indicates that in the random tree T k In order to record the price p i The length of the path required to separate to the leaf node. The shorter the path, the more likely it is that the price record is inconsistent with the overall distribution and tends to be abnormal. avg_path_length(T k ) indicates that in the random tree T k The average separation path length of all samples in is used as the normalized reference value; S i Represents the cumulative relative path score of the i-th price record in K random trees. The larger the sum, the more abnormal the tendency is. ξ is a fixed threshold or offset used to adjust the center position of the Sigmoid function in order to fine-tune the anomaly score; e represents the natural Euler constant; α i Represents the final abnormality; when α i When it is greater than the preset warning value, it means that the price has changed significantly at that moment and requires further attention.

6. A market analysis system based on big data according to claim 5, characterized in that: The linkage optimization algorithm performed by the optimization and feedback module comprises the following steps: S1, receiving the trend prediction result price prediction value output by the time series prediction module, and the abnormality degree output by the abnormality and risk detection module; S2. Combine the two into a comprehensive indicator through statistical mapping to measure the performance of a single price in terms of both trend and potential risk; S3. Preset hyperparameters, wherein the hyperparameters include risk sensitivity coefficients, and dynamically adjust the trust in the price forecast value according to the magnitude of the abnormality; When the abnormality is significantly higher than the risk threshold, excessive trust in price forecasts is suppressed and risk warnings are highlighted; When the degree of abnormality is within the normal range, the trust in the price forecast value is enhanced to improve the accurate prediction of normal market price fluctuations; S4. Compare the integrated comprehensive indicators with the real prices, or accept online feedback from external business systems; S5. Input the comparison results back into the time series prediction algorithm and anomaly detection algorithm to update their network weights or path division strategies, and continuously enhance the system's ability to adapt to changes in the price environment.

7. The market analysis system based on big data according to claim 6 is characterized in that: The price prediction value of the trend prediction result and the abnormality of the risk detection value are integrated through the following formula to output the final comprehensive index: z i =y i ×(1−β×α i )+γ×α i ; y i Represents the predicted value of the price at the i-th moment by the time series prediction algorithm; α i Represents the risk detection value of the anomaly and risk detection algorithm for the i-th price record; β is an adjustable hyperparameter used to control the suppression weight of the risk detection value on the prediction result; The larger the β, the stronger the inhibitory effect is when the degree of abnormality is high; γ is an adjustable hyperparameter used to control the compensation or correction of the risk detection value to the prediction result; the larger γ is, the more significant the abnormality is. i The greater the contribution to the final result; z i It represents the comprehensive price index generated after fusion, which can reflect the correction results of sudden market fluctuations or abnormal risks. This indicator takes into account both trend prediction and abnormal risks, and is ultimately output to the visualization and external interface unit or business system.

8. A market analysis method based on big data, characterized in that: Including using the market situation analysis system based on big data as described in any one of claims 1-7.

Images