Market fluctuation economic management risk prediction system based on big data analysis

Abstract

The invention belongs to the field of risk prediction, and particularly discloses a market fluctuation economic management risk prediction system based on big data analysis, and the system comprises a data collection and preprocessing module, a big data storage module, a risk assessment model construction module, a risk prediction module and an early warning module. According to the method, dimension reduction and feature extraction are performed on preprocessed data by using a method based on principal component analysis and information gain, redundant data are effectively removed, and the method helps to understand complex market related data; constructing a risk prediction model, combining the capturing capability of an autoregression condition heterovariance model for fluctuation characteristics and the strong processing capability of a long and short-term memory network model for a time sequence, and introducing a root type function to convert polar left bias distribution in a label into rightward moving volume increase distribution; the long-short-term memory network is optimized by using a crow search algorithm, so that the dynamic change and nonlinear relationship of the risk can be captured more comprehensively.

Description

Technical Field

[0001] The present invention relates to the field of risk prediction, and specifically refers to a market fluctuation economic management risk prediction system based on big data analysis. Background Art

[0002] Market volatility economic management risk prediction is a forward-looking estimation and judgment of the risks that may be caused by market uncertainty, price fluctuations, and changes in the economic situation, helping managers make more reasonable decisions and enabling companies to prepare for risks in advance. Since financial data is usually very high-dimensional and there is multicollinearity between features, it is a challenge to select meaningful features when using machine learning to predict market volatility economic management risks; market fluctuations are time-dependent, and market data usually contain nonlinear relationships. Existing prediction models have problems with improper predictions and poor processing effects. It is necessary to make accurate predictions quickly on real-time data streams, and the prediction model also needs to be updated in a timely manner when the market environment changes. Summary of the invention

[0003] In view of the above situation, in order to overcome the defects of the prior art, the present invention provides a market volatility economic management risk prediction system based on big data analysis. In view of the technical problem that financial data is usually of high dimension and there is multicollinearity between features, when using machine learning to predict market volatility economic management risks, feature selection is relatively difficult, the present invention uses a method based on principal component analysis and information gain to reduce the dimension and extract features of preprocessed data, effectively remove redundant data, and help the prediction model understand complex market-related data; in view of the time dependence of market fluctuations and the fact that market data usually contains nonlinear relationships, the existing prediction models have the problems of improper prediction and poor processing effect, and it is necessary to make accurate predictions quickly on real-time data streams, and the prediction model also needs to update the model in time when the market environment changes. The present invention constructs a risk prediction model, which combines the ability of the autoregressive conditional heteroskedasticity model to capture volatility characteristics and the powerful processing ability of the long short-term memory network model for time series, and introduces a root function to convert the extreme left-skewed distribution in the label into a volume-increasing distribution moving to the right, and uses the crow search algorithm to optimize the long short-term memory network to more comprehensively capture the dynamic changes and nonlinear relationships of risks.

[0004] The technical solution adopted by the present invention is as follows: The present invention provides a market volatility economic management risk prediction system based on big data analysis, and the market volatility economic management risk prediction system based on big data analysis includes a data acquisition and preprocessing module, a big data storage module, a risk assessment model construction module, a risk prediction module and an early warning module;

[0005] The data collection and preprocessing module collects market-related data, including macroeconomic data, industry data, and market transaction data, and preprocesses the market-related data, including removing duplicate data and obviously erroneous data, and filling missing values ​​with the median to obtain preprocessed data;

[0006] The big data storage module builds a distributed file system for storing all data collected and generated in the data collection and preprocessing module and the risk assessment model construction module;

[0007] The risk assessment model building module uses a method based on principal component analysis and information gain to reduce the dimension and extract features of the preprocessed data, and mixes the autoregressive conditional heteroskedasticity model and the long short-term memory network model to build a risk prediction model;

[0008] The risk prediction module inputs the preprocessed data into the risk prediction model for prediction to obtain a predicted risk size, wherein the risk size is represented by [0, 4], where 0 represents the minimum risk and 4 represents the maximum risk;

[0009] The early warning module issues an alarm message when the risk level is greater than or equal to 3.

[0010] The risk assessment model building module uses a method based on principal component analysis and information gain to reduce the dimension and extract features of the preprocessed data. The method based on principal component analysis and information gain specifically includes the following steps:

[0011] Step A1: Collect a historical market volatility risk dataset, wherein the historical market volatility risk dataset includes a feature set and corresponding labels, wherein the feature set includes historical macroeconomic data, historical industry data, and historical market transaction data, and the label is risk intensity, which is represented by [0, 4], wherein the number 0 represents the minimum risk and the number 4 represents the maximum risk, and the historical market volatility risk dataset is divided into a training set and a test set at a ratio of 3:1;

[0012] Step A2: Calculate the average value of each feature in the feature set, normalize the features, and obtain normalized data;

[0013] Step A3: Calculate the covariance matrix of the normalized data to represent the dependency between features, perform spectral decomposition on the covariance matrix, and obtain a low-dimensional feature set;

[0014] Step A4: Randomly select a feature and calculate the entropy without the feature, which is recorded as the pre-entropy. The formula used is as follows: ;

[0015] In the formula, is the entropy without this feature, It's a label. is a probability distribution, Yes traversal;

[0016] Step A5: Calculate the entropy without removing the feature, recorded as the whole entropy, and the formula used is as follows: ;

[0017] In the formula, is the entropy without removing the feature, It is a feature, Yes traversal;

[0018] Step A6: Repeat steps A4 and A5 to calculate the front entropy and the whole entropy of each feature, calculate the difference between the whole entropy and the front entropy, and obtain the information gain of each feature until all features are traversed;

[0019] Step A7: Preset an information gain threshold and remove features whose information gain is less than the information gain threshold.

[0020] The risk assessment model building module uses a hybrid prediction method to predict the risk size, and the hybrid prediction method specifically includes the following steps:

[0021] Step B1: Establish and initialize the autoregressive conditional heteroskedasticity model and the long short-term memory network model as risk prediction models, and use the root function to transform the distribution of labels from an extremely left-skewed distribution to a volume-increasing distribution that moves to the right. The formula used is as follows: ;

[0022] In the formula, and is a random feature, and They are and The probability density function of , is a very left-skewed distribution in the labels;

[0023] Step B2: input the training set into the risk prediction model for training, using a rolling window method for training;

[0024] Input the test set into the risk prediction model and define the loss function of the risk prediction model. The formula used is as follows: ;

[0025] In the formula, is the loss function, is the number of estimated parameters in the risk prediction model, is the maximum value of the likelihood function of the risk prediction model, is the number of samples in the historical market volatility risk dataset;

[0026] Step B3: Use the crow search algorithm to optimize the long short-term memory network;

[0027] Furthermore, step B3 specifically includes the following steps:

[0028] Step B31: Initialize the position and memory of the crow to obtain the initial position of the crow. In the first iteration, the memory of the crow is the same as the initial position;

[0029] Step B32: Calculate the objective function of each crow as the fitness to iteratively update the position of the crow. The formula used is as follows: ;

[0030] In the formula, and Respectively represent The crow and A crow, is the number of iterations, It is The first crow The updated position, It is The fitness of a crow, It is memory;

[0031] Step B33: If the crow can quickly locate itself by position without relying on memory, then when the crow's position is better than its memory, update the crow's memory, and continuously update the crow's position and memory until the optimal objective function is found.

[0032] The beneficial effects achieved by the present invention using the above scheme are as follows:

[0033] (1) In view of the technical problem that feature selection is difficult when using machine learning to predict market volatility and economic management risks, since financial data is usually highly dimensional and there is multicollinearity between features, the present invention uses a method based on principal component analysis and information gain to reduce the dimension and extract features of preprocessed data, effectively remove redundant data, and help the prediction model understand complex market-related data;

[0034] (2) Market fluctuations are time-dependent and market data usually contain nonlinear relationships. Existing prediction models have problems of improper prediction and poor processing effects. It is necessary to make accurate predictions quickly on real-time data streams. The prediction model also needs to update the model in time when the market environment changes. The present invention constructs a risk prediction model that combines the ability of the autoregressive conditional heteroskedasticity model to capture volatility characteristics and the powerful processing ability of the long short-term memory network model for time series. At the same time, a root function is introduced to convert the extremely left-skewed distribution in the label into a volume-increasing distribution that moves to the right. The crow search algorithm is used to optimize the long short-term memory network to more comprehensively capture the dynamic changes and nonlinear relationships of risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A module connection diagram of a market volatility economic management risk prediction system based on big data analysis provided by the present invention.

[0036] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0038] Example 1: See Figure 1 , this embodiment provides a market volatility economic management risk prediction system based on big data analysis, the market volatility economic management risk prediction system based on big data analysis includes a data acquisition and preprocessing module, a big data storage module, a risk assessment model construction module, a risk prediction module and an early warning module;

[0039] The data collection and preprocessing module collects market-related data, including macroeconomic data, industry data, and market transaction data, and preprocesses the market-related data, including removing duplicate data and obviously erroneous data, and filling missing values ​​with the median to obtain preprocessed data;

[0040] The big data storage module builds a distributed file system for storing all data collected and generated in the data collection and preprocessing module and the risk assessment model construction module;

[0041] The risk assessment model building module uses a method based on principal component analysis and information gain to reduce the dimension and extract features of the preprocessed data, and mixes the autoregressive conditional heteroskedasticity model and the long short-term memory network model to build a risk prediction model;

[0042] The risk prediction module inputs the preprocessed data into the risk prediction model for prediction to obtain a predicted risk size, wherein the risk size is represented by [0, 4], where 0 represents the minimum risk and 4 represents the maximum risk;

[0043] The early warning module issues an alarm message when the risk level is greater than or equal to 3.

[0044] Example 2: See Figure 1 This embodiment is based on the above embodiment. The risk assessment model building module uses a method based on principal component analysis and information gain to reduce the dimension and extract features of the preprocessed data. The method based on principal component analysis and information gain specifically includes the following steps:

[0045] Step A1: Collect a historical market volatility risk dataset, wherein the historical market volatility risk dataset includes a feature set and corresponding labels, wherein the feature set includes historical macroeconomic data, historical industry data, and historical market transaction data, and the label is risk intensity, which is represented by [0, 4], wherein the number 0 represents the minimum risk and the number 4 represents the maximum risk, and the historical market volatility risk dataset is divided into a training set and a test set at a ratio of 3:1;

[0046] Step A2: Calculate the average value of each feature in the feature set, normalize the features, and obtain normalized data;

[0047] Step A3: Calculate the covariance matrix of the normalized data to represent the dependency between features, perform spectral decomposition on the covariance matrix, and obtain a low-dimensional feature set;

[0048] Step A4: Randomly select a feature and calculate the entropy without the feature, which is recorded as the pre-entropy. The formula used is as follows: ;

[0049] In the formula, is the entropy without this feature, It's a label. is a probability distribution, Yes traversal;

[0050] Step A5: Calculate the entropy without removing the feature, recorded as the whole entropy, and the formula used is as follows: ;

[0051] In the formula, is the entropy without removing the feature, It is a feature, Yes traversal;

[0052] Step A6: Repeat steps A4 and A5 to calculate the front entropy and the whole entropy of each feature, calculate the difference between the whole entropy and the front entropy, and obtain the information gain of each feature until all features are traversed;

[0053] Step A7: Preset an information gain threshold and remove features whose information gain is less than the information gain threshold.

[0054] Through the above operations, in order to address the technical problem that feature selection is difficult when using machine learning to predict economic management risks of market fluctuations because financial data is usually very high in dimension and there is multicollinearity between features, the present invention uses a method based on principal component analysis and information gain to reduce the dimension and extract features of preprocessed data, effectively remove redundant data, and help the prediction model understand complex market-related data.

[0055] Example 3: See Figure 1 In this embodiment, based on the above embodiment, the risk assessment model building module uses a hybrid prediction method to predict the risk size. The hybrid prediction method specifically includes the following steps:

[0056] Step B1: Establish and initialize the autoregressive conditional heteroskedasticity model and the long short-term memory network model as risk prediction models, and use the root function to transform the distribution of labels from an extremely left-skewed distribution to a volume-increasing distribution that moves to the right. The formula used is as follows: ;

[0057] In the formula, and is a random feature, and They are and The probability density function of , is a very left-skewed distribution in the labels;

[0058] Step B2: input the training set into the risk prediction model for training, using a rolling window method for training;

[0059] Input the test set into the risk prediction model and define the loss function of the risk prediction model. The formula used is as follows: ;

[0060] In the formula, is the loss function, is the number of estimated parameters in the risk prediction model, is the maximum value of the likelihood function of the risk prediction model, is the number of samples in the historical market volatility risk dataset;

[0061] Step B3: Use the crow search algorithm to optimize the long short-term memory network.

[0062] Example 4: See Figure 1 This embodiment is based on the above embodiment, and step B3 specifically includes the following steps:

[0063] Step B31: Initialize the position and memory of the crow to obtain the initial position of the crow. In the first iteration, the memory of the crow is the same as the initial position;

[0064] Step B32: Calculate the objective function of each crow as the fitness to iteratively update the position of the crow. The formula used is as follows: ;

[0065] In the formula, and Respectively represent The crow and A crow, is the number of iterations, It is The first crow The updated position, It is The fitness of a crow, It is memory;

[0066] Step B33: If the crow can quickly locate itself by position without relying on memory, then when the crow's position is better than its memory, update the crow's memory, and continuously update the crow's position and memory until the optimal objective function is found.

[0067] Through the above operations, market fluctuations are time-dependent, and market data usually contain nonlinear relationships. The existing prediction models have the problems of improper prediction and poor processing effect. It is necessary to make accurate predictions quickly on real-time data streams. The prediction model also needs to update the model in time when the market environment changes. The technical problem is that the present invention constructs a risk prediction model, which combines the ability of the autoregressive conditional heteroskedasticity model to capture volatility characteristics and the powerful processing ability of the long short-term memory network model for time series. At the same time, the root function is introduced to convert the extreme left-skewed distribution in the label into a volume-increasing distribution moving to the right, and the crow search algorithm is used to optimize the long short-term memory network to more comprehensively capture the dynamic changes and nonlinear relationships of risks.

[0068] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0069] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

[0070] The present invention and its embodiments are described above, and such description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by it, without departing from the purpose of the invention, they can design a structure and embodiment similar to the technical solution without creativity, which should belong to the protection scope of the present invention.

Claims

1. A market volatility economic management risk prediction system based on big data analysis, characterized in that: It includes data collection and preprocessing module, big data storage module, risk assessment model building module, risk prediction module and early warning module; The data collection and preprocessing module collects market-related data, including macroeconomic data, industry data, and market transaction data, and preprocesses the market-related data, including removing duplicate data and obviously erroneous data, and filling missing values ​​with the median to obtain preprocessed data; The big data storage module builds a distributed file system for storing all data collected and generated in the data collection and preprocessing module and the risk assessment model construction module; The risk assessment model building module uses a method based on principal component analysis and information gain to reduce the dimension and extract features of the preprocessed data, and mixes the autoregressive conditional heteroskedasticity model and the long short-term memory network model to build a risk prediction model; The risk prediction module inputs the preprocessed data into the risk prediction model for prediction to obtain a predicted risk size, wherein the risk size is represented by [0, 4], where 0 represents the minimum risk and 4 represents the maximum risk; The early warning module issues an alarm message when the risk level is greater than or equal to 3.

2. According to the market volatility economic management risk prediction system based on big data analysis according to claim 1, it is characterized in that: The risk assessment model building module uses a method based on principal component analysis and information gain to reduce the dimension and extract features of the preprocessed data. The method based on principal component analysis and information gain specifically includes the following steps: Step A1: Collect a historical market volatility risk dataset, wherein the historical market volatility risk dataset includes a feature set and corresponding labels, wherein the feature set includes historical macroeconomic data, historical industry data, and historical market transaction data, and the label is risk intensity, which is represented by [0, 4], wherein the number 0 represents the minimum risk and the number 4 represents the maximum risk, and the historical market volatility risk dataset is divided into a training set and a test set at a ratio of 3:1; Step A2: Calculate the average value of each feature in the feature set, normalize the features, and obtain normalized data; Step A3: Calculate the covariance matrix of the normalized data to represent the dependency between features, perform spectral decomposition on the covariance matrix, and obtain a low-dimensional feature set; Step A4: randomly select a feature and calculate the entropy without the feature, which is recorded as the pre-entropy; Step A5: Calculate the entropy without removing the feature, which is recorded as the whole entropy; Step A6: Repeat steps A4 and A5 to calculate the front entropy and the whole entropy of each feature, calculate the difference between the whole entropy and the front entropy, and obtain the information gain of each feature until all features are traversed; Step A7: Preset an information gain threshold and remove features whose information gain is less than the information gain threshold.

3. The market fluctuation economic management risk prediction system based on big data analysis according to claim 2 is characterized in that: The risk assessment model building module uses a hybrid prediction method to predict the risk size, and the hybrid prediction method specifically includes the following steps: Step B1: Establish and initialize the autoregressive conditional heteroskedasticity model and the long short-term memory network model as risk prediction models, and use the root function to transform the distribution of labels from an extremely left-skewed distribution to a volume-increasing distribution that moves to the right. The formula used is as follows: ; In the formula, and is a random feature, and They are and The probability density function of , is a very left-skewed distribution in the labels; Step B2: input the training set into the risk prediction model for training, using a rolling window method for training; Input the test set into the risk prediction model and define the loss function of the risk prediction model. The formula used is as follows: ； In the formula, is the loss function, is the number of estimated parameters in the risk prediction model, is the maximum value of the likelihood function of the risk prediction model, is the number of samples in the historical market volatility risk dataset; Step B3: Use the crow search algorithm to optimize the long short-term memory network.

4. The market fluctuation economic management risk prediction system based on big data analysis according to claim 3 is characterized in that: Step B3 specifically includes the following steps: Step B31: Initialize the position and memory of the crow to obtain the initial position of the crow. In the first iteration, the memory of the crow is the same as the initial position. Step B32: Calculate the objective function of each crow as the fitness to iteratively update the position of the crow. The formula used is as follows: ; In the formula, and Respectively represent The crow and A crow, is the number of iterations, It is The Crow The updated position, It is The fitness of a crow, It is memory; Step B33: If the crow can quickly locate itself by position without relying on memory, then when the crow's position is better than its memory, update the crow's memory, and continuously update the crow's position and memory until the optimal objective function is found.

Images