Data analysis method and system based on artificial intelligence

Abstract

The invention relates to the technical field of data analysis, in particular to a data analysis method and system based on artificial intelligence, and the method comprises the following steps: collecting and arranging variable data of a financial market, the variable data comprising asset price, market transaction volume and policy change information, and obtaining a data integration result; and based on the data integration result, constructing a causal graph by using a Bayesian network, and determining a potential relationship between variables to obtain a causal graph construction result. According to the method, the key factors influencing the financial market variables can be identified and verified, so that the accuracy of model prediction is improved. The Bayesian network helps to construct an intuitive relation graph between variables, and helps to clarify complex interaction between the variables. The introduction of Do calculation further purifies the model, and miscellaneous factors are eliminated by accurately controlling variables, so that the estimation of the causal effect is ensured not to be interfered by external noise.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology, and in particular to a data analysis method and system based on artificial intelligence. Background Art

[0002] The field of data analysis technology covers a wide range of methods and tools for extracting useful information and insights from large amounts of data to support the decision-making process. In the financial field, data analysis is particularly critical because it involves aspects such as the prediction of market trends, risk management, portfolio optimization, and analysis of customer behavior. Financial data analysis uses methods such as statistics, machine learning, econometrics, and time series analysis to process not only structured data such as stock prices and trading volumes, but also unstructured data such as news reports and social media content. However, existing technologies are often limited to direct statistical analysis and simple prediction models when processing financial market data, and lack the exploration of deep causal relationships in the data, which may lead to strategy formulation based on appearances rather than the actual drivers of the market. Summary of the invention

[0003] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a data analysis method and system based on artificial intelligence.

[0004] In order to achieve the above object, the present invention adopts the following technical solution, a data analysis method based on artificial intelligence, comprising the following steps:

[0005] Collect and organize variable data of the financial market, including asset prices, market transaction volume and policy change information, to obtain data integration results; based on the data integration results, use the Bayesian network to construct a causal graph, determine the potential relationship between variables, and obtain a causal graph construction result;

[0006] Based on the result of the causal diagram construction, the causal effect of the target financial variable is derived by applying the Do calculus, and the interference caused by confounding factors is eliminated to obtain the purified causal effect; based on the purified causal effect, the instrumental variables are selected and analyzed, and the validity of the instrumental variables is determined by generalized moment estimation to obtain the instrumental variable analysis results;

[0007] Using the instrumental variable analysis results, the double difference model is used to eliminate non-policy related noise, quantify the impact of policy changes on the asset market, and obtain policy impact assessment results; based on the policy impact assessment results, the policy impact in each time window is judged to obtain dynamic policy assessment results;

[0008] Based on the dynamic policy evaluation results, the time series model is used to predict asset price trends. At the same time, referring to the causal variable input, an asset price and risk prediction model is constructed.

[0009] Preferably, the steps of obtaining the data integration result are:

[0010] Collect asset prices, market trading volume and policy change information from financial market databases, which are derived from exchange reports, government announcements and market analysis reports to obtain a preliminary data set;

[0011] Based on the preliminary data set, perform data cleaning and preprocessing to remove duplicate records, calibrate data format and resolve missing values ​​to obtain a cleaned data set;

[0012] According to the cleaned data set, the data is integrated through data fusion to obtain a data integration result.

[0013] Preferably, the steps for obtaining the causal graph construction result are:

[0014] According to the data integration results, nodes and edges are selected to design a Bayesian network, where nodes represent various financial variables, including asset prices and market trading volumes, and edges represent potential causal relationships between variables, thereby obtaining a preliminary causal relationship model;

[0015] Based on the preliminary causal relationship model, the conditional dependencies between nodes are calculated to obtain the probability distribution. The calculation formula is:

[0016]

[0017] Among them, X i represents financial variables, X pa(i) Represents X i The parent node, x i Represents X i The observed value, μ pa(i) and σ pa(i) They are X pa(i) The mean and standard deviation of P(X i ∣X pa(i) ) means that in the parent node X pa(i) Under the condition of X i The probability distribution of

[0018] According to the probability distribution, the structure of the Bayesian network is updated to obtain a causal graph construction result.

[0019] Preferably, the steps of obtaining the purified causal effect are:

[0020] According to the result of the cause-effect diagram construction, the target financial variables and influencing factors are determined, and the Do calculation method is used to determine the direct and indirect impact of each factor on the target variable. The causal path between the variables is obtained by constructing an impact chain, and the preliminary analysis results of the causal effect are obtained;

[0021] Based on the preliminary causal effect analysis results, identify and eliminate the confounding factors that affect the target variable to obtain adjusted causal effect analysis results;

[0022] The adjusted causal effect analysis results are used to recalculate and evaluate the pure causal relationship of the target financial variable to obtain the purified causal effect.

[0023] Preferably, the steps for obtaining the instrumental variable analysis results are:

[0024] Based on the purified causal effects, potential instrumental variables associated with the target financial variable are selected;

[0025] Based on the potential instrumental variables, the relationship strength between each instrumental variable and the target financial variable is evaluated to obtain the estimated parameters. The calculation formula is:

[0026]

[0027] Where W represents the instrumental variable matrix, M is the weight matrix derived from the causal graph model, and Y is the observed value of the target variable. represents the estimated parameters;

[0028] Based on the estimated parameters, the effectiveness of the instrumental variables is evaluated to verify whether the instrumental variables can estimate the causal effect of the target financial variables, and the results of the instrumental variable analysis are obtained.

[0029] Preferably, the steps for obtaining the policy impact assessment results are:

[0030] Based on the results of the instrumental variable analysis, the difference in market reactions before and after the implementation of the policy is calculated to obtain the policy impact. The calculation formula is:

[0031]

[0032] Among them, Y t2 and Y t1 Represent the market performance of the experimental group before and after the policy implementation, C t2 and C t1 Respectively represent the performance of the control group at the same time, N represents the number of observations, and ΔImpact is the policy impact after adjustment for the reference time effect;

[0033] The policy impact quantity is used to evaluate the policy impact, verify whether the policy change has a statistical impact, determine the performance of the policy impact in the asset market, and obtain the policy impact evaluation result.

[0034] Preferably, the steps for obtaining the dynamic policy evaluation results are:

[0035] Based on the policy impact assessment results, obtain market performance data within each time window to form an organizational data set sorted by time window;

[0036] Based on the organizational data set sorted by time windows, identifying the difference in market response before and after the policy change, determining the policy impact of each window by comparing and analyzing the data changes in each time window, and obtaining the time window policy impact analysis results;

[0037] Based on the policy impact analysis results of the time window, the effect of policy changes on the market is evaluated through continuous time series analysis to obtain dynamic policy evaluation results.

[0038] Preferably, the steps for obtaining the asset price and risk prediction model are:

[0039] Based on the dynamic policy evaluation results, the ARIMA model is used to predict asset price trends and obtain asset prices at the prediction time. The formula is:

[0040] P t =α+βP t-1 +∈ t

[0041] Among them, P t represents the asset price at the prediction time, P t-1 represents the asset price at the previous point in time, α and β are model parameters, ∈ t is the error term;

[0042] Based on the asset price at the prediction time, the impact of causal variables is integrated to construct an asset price and risk prediction model.

[0043] The present invention provides a data analysis system, comprising:

[0044] The data collection module organizes the asset prices, market transaction volumes and policy change information of the financial market to obtain an integrated data set;

[0045] The association analysis module uses the integrated data set to analyze the potential relationships between variables, construct a causal graph, and obtain a causal relationship diagram;

[0046] The purification analysis module uses the causal relationship diagram to analyze and eliminate the influence of confounding factors, and applies the double difference model to obtain the purified causal effect results;

[0047] The instrumental variable analysis module uses the causal effect results to select instrumental variables for generalized moment estimation, determine the effectiveness, and obtain the results of the instrumental variable effectiveness analysis;

[0048] The policy impact assessment module uses time series models to predict asset price trends based on the results of instrumental variable effectiveness analysis, builds price and risk prediction models, and obtains dynamic policy evaluation results.

[0049] Compared with the prior art, the advantages and positive effects of the present invention are:

[0050] In the present invention, the key factors that affect financial market variables can be identified and verified, thereby improving the accuracy of model predictions. Bayesian networks help construct intuitive relationship maps between variables, which helps to clarify the complex interactions between variables. The introduction of Do calculus further purifies the model, eliminates confounding factors by precisely controlling variables, and ensures that the estimation of causal effects is not interfered with by external noise. In addition, the use of a double difference model to evaluate the specific impact of policy changes can distinguish policy effects from other market change factors, providing a scientific basis for policy formulation and adjustment. The comprehensive application of this method not only enhances the depth of data processing, but also enhances the data-driven power of decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a schematic diagram of the steps of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0053] See also Figure 1 The present invention provides a technical solution, a data analysis method based on artificial intelligence, comprising the following steps:

[0054] Collect and organize variable data of financial markets, including asset prices, market transaction volume and policy change information, and obtain data integration results; Based on the data integration results, use Bayesian networks to construct causal graphs, determine the potential relationship between variables, and obtain causal graph construction results;

[0055] Based on the results of the causal diagram construction, the Do calculus is used to derive the causal effect of the target financial variable, eliminating the interference caused by confounding factors to obtain the purified causal effect; based on the purified causal effect, instrumental variables are selected and analyzed, and the validity of the instrumental variables is determined through generalized moment estimation to obtain the results of the instrumental variable analysis;

[0056] Using the results of instrumental variable analysis, the double difference model is used to eliminate non-policy related noise, quantify the impact of policy changes on the asset market, and obtain the policy impact assessment results; based on the policy impact assessment results, the policy impact in each time window is judged to obtain dynamic policy assessment results;

[0057] Based on the results of dynamic policy evaluation, time series models are used to predict asset price trends. At the same time, asset price and risk prediction models are constructed with reference to causal variable inputs.

[0058] The steps to obtain the data integration results are:

[0059] Collect asset prices, market trading volume and policy change information from financial market databases, which are derived from exchange reports, government announcements and market analysis reports to obtain a preliminary data set;

[0060] Based on the preliminary data set, perform data cleaning and preprocessing to remove duplicate records, calibrate data format and resolve missing values ​​to obtain a cleaned data set;

[0061] According to the cleaned data set, the data is integrated through data fusion to obtain the data integration result.

[0062] Specifically, after collecting asset prices, market trading volumes and policy change information from the financial market database, it is necessary to read and analyze the original contents from exchange reports, government announcements and market analysis reports one by one, including identifying records related to asset price fluctuations, screening interval data that can reflect changes in market trading volumes, and performing text analysis on policy changes. During the analysis process, policy types can be distinguished by referring to pre-defined domain terms. For example, financial regulatory measures and tax incentives can be classified by name and issuing authority. All collected records are then compared with the pre-established valid range. For example, the price value is judged by the range of 0 to 50,000, and the trading volume is judged by the range of 0 to 100,000,000. Records can be marked as abnormal based on field experience and existing statistical results. This marking method can further identify potential erroneous data or missing data. If some records involve policy release information but the issuing agency is unknown or the release date is not within an acceptable period, they will be marked as incomplete data. These incomplete data will be archived separately to facilitate finding the specific source and confirming the threshold setting method in subsequent links. The abnormal judgment value for price or trading volume is mainly based on the maximum and minimum value range in the market public data statistics of previous years or reference industry yearbooks. When setting the threshold through experience, the threshold can be initialized without exceeding three times the standard deviation of the statistical mean. Finally, after the reading is completed, a comprehensive set of multiple elements including asset prices, trading volumes and policy changes is formed to obtain a preliminary data set.

[0063] Based on the preliminary data set, all records need to be further processed, including checking the records one by one by field, rechecking the data marked as abnormal through the pre-defined correction strategy, and if there are missing values, interpolation and supplementation are performed in combination with records with the same or similar conditions in the same time period. For example, in the transaction volume field, the upper and lower floating ranges of the transaction volumes in adjacent periods of the same day can be compared. If the floating range is between 0 and 5000, the interpolation algorithm is tried to fill the missing value. If it exceeds this range, a more accurate value is determined by consulting reference data. In addition, if there is duplicate content in text-based policy information, it is merged and the version with the most recent timestamp is retained. At the same time, the effective interval verification of the numerical parameters in all records is performed again. For example, the asset price is compared within the range of 0 to 50,000. If extreme values ​​appear, a preset threshold based on the comprehensive statistics of the maximum and minimum values ​​can be set according to field experience for judgment, and it is regarded as a reasonable record when the error does not exceed twice the standard deviation of the mean. The records that cannot be confirmed to be correct are marked and included in the subsequent review process. After all the above integration and supplementation are completed, a new and more complete set will be obtained, and the cleaned data set will be obtained.

[0064] In order to integrate information from different sources through data fusion based on the cleaned data set, it is necessary to establish corresponding comparison relationships in the database. For example, cross-comparison of transaction volumes from different reports on the same date is performed, and the questionable data rows are reconfirmed through the comparison results. At the same time, a set of range criteria is set before fusion to unify inconsistent annotation methods. For example, when the description of the same policy information appears differently in multiple files, it is necessary to make an association judgment based on the release time of the files or the authority of the issuing agency. If there are only slight text differences between the data from multiple sources and the release time is similar, they are regarded as the same policy and merged. If multiple sources are If there is a large deviation in the transaction volume field of the source records, a reference value based on the previous market volatility can be set for judgment. For example, when the historical volatility of the same type of assets does not exceed 10%, the records with deviations exceeding 15% are marked for re-review. All records that can be confirmed or corrected are finally spliced ​​into a unified entry. If the same indicator data conflicts in individual time periods and cannot be corrected, it can be supplemented by interpolation weighted calculation based on the existing industry average or the values ​​of adjacent days in the time period. After the fusion of all fields is completed, a set of data in a unified format with multi-source information complementarity can be output to obtain the data integration result.

[0065] The steps to obtain the causal graph construction results are:

[0066] According to the data integration results, nodes and edges are selected to design the Bayesian network. The nodes represent various financial variables, including asset prices and market trading volumes, and the edges represent the potential causal relationships between variables, thus obtaining a preliminary causal relationship model.

[0067] Based on the preliminary causal relationship model, the conditional dependencies between nodes are calculated to obtain the probability distribution. The calculation formula is:

[0068]

[0069] Among them, X i represents financial variables, X pa(i) Represents X i The parent node, x i Represents X i The observed value, μ pa(i) and σ pa(i) They are X pa(i) The mean and standard deviation of P(X i ∣X pa(i) ) means that in the parent node X pa(i) Under the condition of X i The probability distribution of

[0070] According to the probability distribution, the structure of the Bayesian network is updated to obtain the causal graph construction result.

[0071] Specifically, after selecting nodes and edges to design the Bayesian network based on the data integration results, it is necessary to first screen out nodes with representative and identifiable characteristics from the previously obtained financial variables such as asset prices and market trading volumes, including selecting variables with a high correlation with regional policy risks or macroeconomic indicators in the records, and identifying the type of each node for subsequent distinction when setting the relationship between nodes. Subsequently, it is necessary to read the information on the relationship between these nodes and combine the changing trend of historical data to determine whether to set edges between the nodes. During the determination process, the high and low fluctuations of the asset price series and the market trading volume series at the corresponding time points can be recorded, and each record can be compared with the pre-established valid range. For example, the price series can be compared with the interval of 0 to 50,000. If the intersection The corresponding range of Yiliang is between 0 and 100000000. The records outside this range will be marked for reconfirmation. After confirmation, if a highly coupled fluctuation range is found between price and trading volume in the same time range, a potential causal edge is introduced between the two nodes, and further investigation is carried out in combination with the list of economic events on the trading day or week in which they are located. If the list of economic events covers policies, financial instruments or large corporate behavior information related to these nodes, it is feasible to determine the existence of this edge. In addition, a horizontal comparison of market indicators of the same type can be carried out, such as the linkage analysis of trading volume and stock prices of multiple listed companies in the same industry. If synchronous fluctuations occur in the same event window, edges can be set synchronously between these nodes. Finally, after all nodes and edges are determined, a preliminary causal relationship model is obtained.

[0072] The benefit of the formula is that it uses the mean and standard deviation of the parent node to quantitatively evaluate the conditional probability of the target variable, thereby realizing the measurement of conditional dependence between nodes in the network structure; i The acquisition steps are as follows: by sorting out the specified financial market samples, collecting the closing price of a certain stock as x i , the specific value comes from the daily closing price in the trading system and is recorded in the asset price sequence, X pa(i) The acquisition step is to collect the mean and standard deviation of the influencing factor node of the stock in the causal graph obtained previously, μ pa(i) With σ pa(i) The acquisition steps are to make statistics on the observation values ​​of the influencing factor nodes in the same time window, calculate the mean and standard deviation and store them in the parameter set. For example, set the observation window to the past 60 trading days, add the observation values ​​of the influencing factor nodes and divide them by 60 to get the mean, then divide the sum of squared deviations by 60 to get the variance and take the square root to get the standard deviation.

[0073] Calculation process:

[0074] Read the closing price record of the target stock on a certain day x i = 27.5 yuan, and the parent node mean μ pa(i) =30.2 yuan, standard deviation σ pa(i) =3.1 yuan into the formula;

[0075] Calculation in molecules That is (27.5-30.2) 2 / (2×3.1 2 ), the value is (-2.7) 2 / (2×9.61)=7.29 / 19.22≈0.3795, then find e -0.3795 ≈0.6844;

[0076] Denominator

[0077] Divide the numerator by the denominator to get

[0078] The results show that when the target stock price is 27.5 yuan, under the condition that the parent node impact factor mean is 30.2 yuan and the standard deviation is 3.1 yuan, X i The conditional probability of the observed value occurring is about 0.0881, which means that the deviation of the currently observed price from the parent node influence level is obvious. If the probability value is subsequently compared with the distribution of other nodes, the position of the value in the entire network structure can be determined, as well as the impact on the node connection relationship in the causal graph construction result.

[0079] The steps to obtain the purified causal effect are:

[0080] According to the results of the cause-effect diagram, the target financial variables and influencing factors are determined, and the Do calculation method is used to determine the direct and indirect impact of each factor on the target variable. The causal path between variables is obtained by constructing an impact chain, and the preliminary analysis results of the causal effect are obtained;

[0081] Based on the preliminary analysis results of causal effects, the confounding factors that affect the target variables are identified and eliminated to obtain the adjusted causal effect analysis results;

[0082] Using the adjusted causal effect analysis results, the pure causal relationship of the target financial variable is recalculated and evaluated to obtain the purified causal effect.

[0083] Specifically, according to the causal diagram construction results obtained earlier, the target financial variable and the corresponding influencing factor are determined. It is necessary to cross-browse and compare all the financial variables obtained earlier one by one to clarify which variables have a continuous or periodic joint change relationship with the target financial variable in the historical observation window. Then, these influencing factors are recorded one by one in a list and their observation data sources are identified. For example, the fluctuations in the relevant period can be retrieved from the asset price series and related policy records retained earlier. If a variable has repeatedly shown obvious jumps in the continuous monitoring of the price range of 0 to 50,000, such as an increase or decrease of more than 3% and the cumulative number of occurrences exceeds 15 times, the time points corresponding to these jump positions can be further summarized and compared with other sub-variables to identify whether the variable has a traceable impact. It is also necessary to classify the direct and indirect effects of the same influencing factor, and regard factors that change the pricing of the target financial variable on the same day as direct effects, and regard behaviors that span several trading days or affect other nodes and then transmit to the target financial variable as indirect effects. Then, construct an influence chain for each influencing factor in the list, referring to the association path between the influencing factor and other variables and arranging them in chronological order. If multiple cumulative fluctuations of intermediate variables appear in some chains, the threshold setting method of impact transmission can be marked when recording, such as quantifying the transmission intensity between 0 and 1 and determining the threshold benchmark through the maximum and minimum values ​​appearing in historical observation data. If the transmission intensity of an influencing factor is higher than 0.5, it is considered to be significantly associated. After completing the sorting of all chains, the preliminary analysis results of the causal effect are obtained.

[0084] Based on the preliminary analysis results of the causal effect obtained above, we can check and eliminate the confounding factors that affect the target financial variables. We need to check the direct and indirect effects previously identified one by one, and determine whether there are situations where multiple key indicators are associated at the same time by retrieving the multiple effects of these influencing factors in historical data. If it is found that a certain influencing factor also forms a set of strong correlations with other variables when it changes at the same time as the target financial variable, it will be regarded as a possible confounding factor, and the degree of coupling between the factor and the target financial variable on the same trading day or even across trading days will be quantified, such as comparing the measured indicator with variables of different dimensions at the same time, and recording its distribution range at different time points. To determine whether it exceeds the preset threshold, such preset threshold can be set to construct a filtering interval based on the sum of the standard deviations of the existing transaction data and combined with the minimum value. For example, when the interval is between 0 and 10, it indicates that the influencing factor has no significant deviation within the investigation range. If its coupling value exceeds this interval, it may produce confounding characteristics. These confounding factors are then uniformly marked and temporarily excluded in subsequent calculations. The statistical information of the remaining influencing factors is re-called to compare the time difference and numerical difference between the observed value and the target financial variable one by one, and further check whether there is any cross-series implicit confounding. After eliminating them together, a new set of influencing factors can be obtained, and the adjusted causal effect analysis results can be obtained.

[0085] Using the adjusted causal effect analysis results obtained above, the pure causal relationship of the target financial variable is recalculated and evaluated. The causal path that has been eliminated from confounding factors needs to be reloaded and the strength of the correlation between each factor and the target financial variable is checked one by one. The high-frequency data recorded by the actual market performance can be combined to evaluate whether the factor continues to intervene in the price change process of the target financial variable within the pre-set observation range, and compare the number of times the factor occurs and the degree of fluctuation. For example, when the number of times the factor occurs is between 10 and 20 and the corresponding price fluctuation is between 1% and 3%, it is determined that this factor has an impact on the price change process of the target financial variable. The target financial variable has potential driving force. If the number of effects exceeds 20 and the corresponding price fluctuation is cumulatively greater than 3%, it is considered a key impact. At the same time, the quantified causal relationship is numerically remapped, and the mapping results are written into the node structure of the causal chain to form a set of traceable paths again. It is further verified whether the numerical continuity in the path is consistent with the fluctuation cycle of the target financial variable. If the degree of consistency can reach a certain empirical threshold, it is considered that the influencing factor and the target financial variable have constructed a strong connection path. After completing the above multi-layer verification, the comprehensive impact of each retained factor on the target financial variable can be clarified to obtain a purified causal effect.

[0086] The steps to obtain the results of instrumental variable analysis are as follows:

[0087] Select potential instrumental variables associated with the target financial variable based on the purified causal effect;

[0088] Based on the potential instrumental variables, the relationship strength between each instrumental variable and the target financial variable is evaluated to obtain the estimated parameters. The calculation formula is:

[0089]

[0090] Where W represents the instrumental variable matrix, M is the weight matrix derived from the causal graph model, and Y is the observed value of the target variable. represents the estimated parameters;

[0091] Based on the estimated parameters, the effectiveness of the instrumental variables is evaluated to verify whether the instrumental variables can estimate the causal effect of the target financial variables and obtain the results of the instrumental variable analysis.

[0092] Specifically, based on the purification causal effect, after selecting the potential instrumental variables associated with the target financial variable, it is necessary to first retrieve the purification causal effect analysis results obtained previously, and compare the influencing factors involving the target financial variable one by one, to confirm which influencing factors have relatively stable mutual independence and numerical correlation with the target financial variable, and record these influencing factors as candidates for potential instrumental variables. When recording, their specific sources and corresponding numerical ranges can be broken down. For example, in the transaction information, a certain macroeconomic indicator can be monitored in the numerical range of 0 to 100, and abnormal records exceeding 100 or below 0 can be marked after comparing with past observation conclusions again. It is also necessary to review whether other economic variables in the same observation period have a coupling relationship with the indicator. When it is determined that the indicator is between 0 and 100, When the mean value exceeds the specified threshold, it is necessary to quantitatively identify it in combination with the historical mean reference obtained previously. The setting of the threshold can be based on segmented statistics of data from the past year or longer period, and the interval with the highest frequency of occurrence will be used as the main reference interval to define the available range of the indicator as an instrumental variable. If the above conditions are met, it will be included in the list of potential instrumental variables, and the specific numerical trajectory of the indicator over time will be recorded in subsequent analysis. By comparing each candidate variable with the fluctuation of the target financial variable in the same period, refer to high-frequency or low-frequency observation data to extract their matching degree at several observation points and record the matching results. If the number of matching results within a certain interval exceeds a certain value, it is judged that there is an association with the target financial variable. After all candidate variables are compared, the potential instrumental variable associated with the target financial variable is obtained.

[0093] The benefit of the formula is that by integrating the instrumental variables, weight matrix and target variable observations in matrix form, the causal estimation of the target financial variable can obtain more systematic coefficients among multiple interference factors, which is convenient for subsequent quantification and testing.

[0094] The steps for obtaining the W parameter are to summarize the previously selected instrumental variables into column vectors one by one, and arrange the observations in different time periods in rows to form a matrix. For example, the daily values ​​of two candidate instrumental variables are monitored for 30 consecutive trading days to obtain a 30×2 matrix.

[0095] The steps to obtain the M parameter are to collect the weight information corresponding to each instrumental variable according to the causal graph model obtained above, and quantify the importance of different factors. If an instrumental variable appears frequently in historical statistics and has a high correlation, it will have a larger weight in the weight matrix. These weights can be obtained by performing segmented statistics on historical observations and calculating the frequency of occurrence, and then filling them into the M matrix according to the corresponding positions.

[0096] The steps for obtaining the Y parameter are to read the observations matching the above 30 trading days one by one from the historical data of the target financial variable, and construct them into a 30×1 vector in the form of a column vector. During the reading process, records with obvious distortion or missing information must be excluded, and the remaining complete observations constitute Y;

[0097] Calculation process:

[0098] Define W as a 30×2 matrix. Take two candidate instrumental variables for the first 30 trading days, and their values ​​are recorded to form:

[0099]

[0100] Define M as a 2×2 weight matrix. For example, in the previous analysis, a certain instrumental variable appears more frequently, so the weight coefficient of this variable is relatively large, and we get:

[0101]

[0102] Define Y as a 30×1 vector to store the observed values ​​of the target financial variable within 30 days, for example:

[0103]

[0104] Calculate W T MW, first multiply W and M to get a 30×2 matrix, and then multiply W T Multiply to form a 2×2 matrix and take its inverse;

[0105] Calculate W T MY, get a 2×1 vector;

[0106] Multiply the results of the previous step and the step before that to get

[0107] In a complete example, if W T The calculation result of MW is Its inverse matrix is At the same time, if W T MY but:

[0108]

[0109] This result shows that when the values ​​of the instrumental variable in 30 trading days and the distribution of the target financial variable are as shown in the above example, the estimated parameters In the two dimensions, they are approximately 7.36 and 6.73 respectively. Different numerical results can indicate the degree of explanation and impact of different instrumental variables on the target financial variables. When the value of a certain dimension is relatively larger, it indicates that the influence of the instrumental variable on the target financial variable is relatively stronger.

[0110] After evaluating the effectiveness of the instrumental variables based on the estimated parameters, we need to The numerical values ​​correspond to the historical fluctuation trends of the target financial variables one by one, and record whether the estimated strength of each instrumental variable is stable in the corresponding observation period. The observed values ​​of the instrumental variables are paired with the daily price or trading volume data of the target financial variables previously maintained for analysis. Each paired record must also be range-checked. For example, the values ​​of the instrumental variables are screened between 0 and 5, and the price range of the target financial variable is compared between 0 and 50,000. If an instrumental variable maintains the same trend of change as the target financial variable in multiple interval comparisons and the amplitude meets the threshold set in advance based on historical data, it is determined that it has a measurable causal relationship with the target financial variable. In this process, daily data from the past year or longer can be used to measure the time-lagged impact of the instrumental variable on the target financial variable. If the time lag is too long or the correlation amplitude deviates significantly, the instrumental variable can be considered temporarily unqualified. Finally, a list of all qualified instrumental variables that have an obvious causal relationship with the target financial variable is summarized and their corresponding The numerical values ​​were cross-compared to obtain the results of instrumental variable analysis.

[0111] The steps to obtain the policy impact assessment results are as follows:

[0112] Based on the results of instrumental variable analysis, the market reaction difference before and after the policy implementation is calculated to obtain the policy impact. The calculation formula is:

[0113]

[0114] Among them, Y t2 and Y t1 Represent the market performance of the experimental group before and after the policy implementation, C t2 and C t1Respectively represent the performance of the control group at the same time, N represents the number of observations, and ΔImpact is the policy impact after adjustment for the reference time effect;

[0115] Use the policy impact metric to evaluate the policy impact, verify whether the policy change has a statistical impact, determine how the policy impact performs in the asset market, and obtain the policy impact assessment results.

[0116] Specifically, the benefit of the formula is that by accumulating the difference between the experimental group and the control group after and before the implementation of the policy under the same number of observations N, a difference value adjusted for time effects can be obtained, thereby identifying the impact of the policy intervention in the target market.

[0117] Y t2 The acquisition steps are as follows: observe the target market indicator in the experimental group sample, record the indicator daily for N consecutive days after the policy is implemented, obtain a set of later observation value sequences and quantify each observation value uniformly. For example, select the average transaction amount of a certain stock as an indicator, read the transaction amount data of the day in the trading system daily, and form {Y 1,t2 ,Y 2,t2 ,…,Y N,t2};

[0118] Y t1 The acquisition steps are as follows: the same experimental group samples as above, read the target market index daily for N consecutive days before the policy implementation to form {Y 1,t1 ,Y 2,t1 ,…,Y N,t1}, and ensure that the length of the natural day or statistical period corresponds to the observation period after implementation;

[0119] C t2 The steps to obtain the index are to select similar market entities or related samples that are not directly affected by the policy in the control group, and record the index formation data N days after the policy is implemented. 1,t2 ,C 2,t2 ,…,C N,t2}, for example, the average stock transaction amount is also selected, but it is necessary to confirm that this control group has not been subject to policy intervention;

[0120] C t1 The steps to obtain the index are as follows: the same control group sample as above is read daily N days before the policy is implemented to form {C 1,t1 ,C 2,t1 ,…,C N,t1};

[0121] The steps to obtain N are to determine an observation period and record it continuously. For example, in the trading day dimension, 20 trading days can be selected as an observation window, and the corresponding number of observations is 20. Data is collected uniformly within the time range selected in the previous article.

[0122] Calculation process:

[0123] Calculate ∑(Y t2 -Y t1 ): For example, within 20 trading days, the transaction amount series of the experimental group is subtracted and then summed. If the sum of the differences is 2 million;

[0124] Calculate ∑(C t2 -C t1 ): Also within 20 trading days, the difference of the control group sequence is taken and then summed. If the sum of the difference is 1.2 million;

[0125] Divide each of the above two terms by N=20, and we get and Assume that the two are 100,000 and 60,000 respectively;

[0126] Subtract according to the formula, that is, ΔImpact = 100,000 - 60,000 = 40,000;

[0127] The results show that after comparing the differences with the control group and normalizing them, the change in transaction volume in the experimental group after the implementation of the policy was 40,000 higher than before the implementation. If this value is much larger than the upper limit of normal fluctuations over a period of time in subsequent statistical tests, it means that the policy does have a greater impact on the target market. If the value is close to 0, it means that the policy impact is relatively small.

[0128] When using the policy impact to evaluate the policy impact, it is necessary to refer to and compare the ΔImpact value obtained in the previous step with the average floating level of the market in a wider historical period or other similar markets one by one, and combine the data characteristics of the experimental group and the control group in different periods recorded previously to identify potential deviations. It is necessary to read the transaction volume indicators or price indicators of the experimental group and the control group on a daily basis, and compare these indicators with the pre-established reasonable ranges. For example, in terms of transaction volume, compare the range of 0 to 100,000 million. When comparing, first check whether the numerical distribution is within the normal range. If it is found that any data point is significantly lower than 0 or higher than 100,000, then record these deviated samples, and re-check or eliminate these deviated samples, and then sort the remaining data according to the time before and after the implementation of the policy. Compare them sequentially to see whether the increase or decrease value of each trading day exceeds the threshold value based on historical volatility. The threshold value can be formed by selecting the mean plus three times the standard deviation of the daily volatility data of the same indicator in the past year. For example, the threshold value is 5%. When the increase or decrease value of a trading day exceeds 5%, it is recorded as an abnormal floating value. Then all abnormal floating values ​​are compared with the abnormal floating values ​​of the control group. In this way, the interval or amplitude that may be amplified by the policy impact can be found. After confirming all the observations, the dates whose values ​​are within a reasonable threshold range and show a relatively consistent trend are aggregated and analyzed together with the ΔImpact calculated previously. Finally, after comparative analysis, the changing trend or amplitude range of the policy impact in the asset market is determined to obtain the policy impact assessment results.

[0129] The steps to obtain dynamic policy evaluation results are as follows:

[0130] Based on the policy impact assessment results, obtain the market performance data within each time window to form an organizational data set sorted by time window;

[0131] Based on the organizational data set sorted by time window, the differences in market responses before and after policy changes are identified. By comparing and analyzing the data changes in each time window, the policy impact of each window is determined, and the policy impact analysis results of the time window are obtained;

[0132] Based on the results of the time window policy impact analysis, through continuous time series analysis, the effect of policy changes on the market is evaluated to obtain dynamic policy evaluation results.

[0133] Specifically, according to the policy impact assessment results obtained above, it is necessary to screen the specific policy implementation time points involved in the assessment results one by one, and set up an independent window span for each time point, and aggregate and record the market performance data in the form of days, weeks or months. When aggregating, it is necessary to make the data from different sources consistent. For example, for price-type indicators, first determine their effective range, such as limiting the stock price range to between 0 and 50,000. If there are records below 0 or above 50,000, they will be marked and re-checked. For volume-type data, you can also set an interval between 0 and 100,000,000 for comparison. If the interval If obvious abnormal fluctuations occur, they will be synchronously checked in combination with the relevant indicators obtained previously. All confirmed valid data entries will be sorted according to their corresponding time spans and assigned to their respective time windows. If it is found that some observation periods overlap with other windows, it is necessary to judge based on the policy effective time or stop time point indicated in the policy impact assessment results obtained previously. The data that overlaps across windows will be split and included in the window that best reflects the changes in the data, so as to construct a relatively complete market performance sequence in each window. Finally, all windows and their corresponding market performance data will be combined to form an organized data set sorted by time windows.

[0134] Based on the organizational data set sorted by time window, it is necessary to retrieve the previously sorted policy implementation period and its effective range. By searching and extracting the fluctuation of market indicators in each time window, the numerical difference of the same indicator before and after the policy change is compared. If the numerical difference exceeds the range obtained in advance, the abnormal fluctuation of the window is recorded. The range can be set by selecting the mean of transaction amount or price fluctuation from historical data plus a number of standard deviations. For example, the threshold is set to the mean plus three times the standard deviation. When the fluctuation value in a window exceeds this threshold, it is considered a significant change. Then, the significant change items of all windows are merged for review. If there are multiple significant changes in multiple windows, the significant change items are combined for review. If significant changes of the same type are detected in adjacent windows, it means that there is still a lasting impact in the later period of time after the policy change. Multi-dimensional parameters can be used to judge the size of fluctuations during comparative analysis, such as reference price, trading volume, number of transactions, etc., and these parameters can be compared according to their respective effective ranges to determine the degree of fluctuation. Once it is confirmed that the fluctuation value of a certain window is significantly higher or lower than the historical average range, the strong correlation of the window with the policy change is recorded. After completing the analysis of all windows, a corresponding policy impact label is assigned to each window, and the overall difference distribution in each time period is aggregated based on these labels to obtain the policy impact analysis results of the time window.

[0135] Based on the results of the policy impact analysis of the time window, it is necessary to conduct continuous time series analysis on the windows marked with the policy impact label, connect the previously obtained changes in key indicators such as price and trading volume in each window into a complete sequence, and conduct longitudinal inspection on the data of each time period in combination with the previously defined observation interval. During the inspection, the price parameter can be set between 0 and 50,000, and the trading volume parameter can be set between 0 and 100,000,000 for line-by-line comparison. When there is an obvious rise or fall in several consecutive windows and the amplitude exceeds a set threshold, for example, the threshold is determined by the floating rate and average standard deviation of the data in the past year, it will be recorded and combined with the previous policy impact label to check whether this fluctuation is closely related to the policy change. If it is detected that some time windows form a progressive impact, it means that the policy has produced a phased distribution change at different stages. These stages can be independently segmented in the sequence analysis and the numerical distribution obtained by this segmentation can be summarized. By cross-comparing the segmented subsequences with the subsequences of other windows to confirm whether there is a repetitive or continuous phenomenon, after the whole process is completed, a comprehensive judgment on the effect of policy changes in the market over a long span is obtained, and the dynamic policy evaluation results are obtained.

[0136] The steps to obtain the asset price and risk prediction model are:

[0137] Based on the results of dynamic policy evaluation, the ARIMA model is used to predict asset price trends and obtain asset prices at the prediction time. The formula is:

[0138] P t =α+βP t-1 +∈ t

[0139] Among them, P t represents the asset price at the prediction time, P t-1 represents the asset price at the previous point in time, α and β are model parameters, ∈ t is the error term;

[0140] Based on the asset price at the prediction time, the impact of causal variables is integrated to build an asset price and risk prediction model.

[0141] Specifically, the formula is beneficial in that it associates the price P at the previous point in time from a time series perspective. t-1 , and superimpose the constant term α that can be monitored or calculated and the coefficient β that can change with the market dynamics, which can be used to predict the asset price P at the current time. t Estimating trend and quantifying random fluctuations∈ t impact.

[0142] The steps to obtain the α parameter are to test the long-term mean of the target asset in the specified observation period, add up the daily closing prices in the observation period and divide it by the number of observation days to get the mean, and make corrections within a certain range based on the volatility data obtained above. The specific corrections can be provided by field experience or statistical results. For example, if the average closing price for 60 consecutive days is 28.5, and historical data show that the benchmark fluctuation range of asset prices in the economic cycle is 3 yuan, then the mean plus an appropriate correction value can be extracted to form α;

[0143] The steps to obtain the β parameter are to read the historical data series of the price at the previous time point and the price at the current time point, and use the time series fitting method obtained above to obtain the correlation coefficient between the two. The value can be derived from statistical regression analysis. If it is found that the price increases by 1 yuan in the previous time point on average by 0.75 yuan at the next time point, then β≈0.75. In actual implementation, it is necessary to make limited corrections to the local effects brought about by the adjustment of the economic environment or the implementation of new policies.

[0144] P t-1 The steps to obtain the parameters are to directly read the asset price from the record of the previous day or the previous observation period. If the analysis is set to be based on the daily closing price, then P t-1 It is the closing price of the previous trading day. For example, P is obtained in the price collection on a certain day. t-1 =27.2 yuan;

[0145] ∈ t The steps to obtain the parameters are to retrieve the remaining fluctuations that are not explained by α and β in the same period, estimate the mean of these residuals to be 0 and regard them as a set of random errors. The specific values ​​are obtained through the residual equation or white noise detection method in the time series model. For example, after statistical analysis, it is confirmed that ∈ t The standard deviation of the residuals is about 0.8, and these residuals behave differently at each time point.

[0146] Calculation process:

[0147] Select the price P of the previous day t-1 =27.2;

[0148] Set α = 28.5 as the constant term after the previous mean correction;

[0149] Substitute the β≈0.75 obtained above into the equation;

[0150] For ∈ t , set it as a random item with a residual mean of 0 and a standard deviation of 0.8, at a specific time point∈ t It fluctuates within the standard deviation range. For example, t = +0.5;

[0151] Substitute the above parameters into the formula one by one:

[0152] P t =28.5+0.75×27.2+0.5

[0153] First calculate βP t-1 = 0.75 × 27.2 = 20.4, then add the result to α = 28.5, and then add ∈ t =0.5, we get:

[0154] P t =28.5+20.4+0.5=49.4

[0155] The result shows that at the current point in time, based on the analysis of the price at the previous point in time and the long-term mean correction term, the predicted asset price is approximately 49.4 yuan. If subsequent observations find that the actual price on a certain day is higher than 50 or lower than 45, the abnormal fluctuations of the residual term can be further checked, and the deviation range can be analyzed in combination with the dynamic policy evaluation results obtained previously or important industry news events.

[0156] Based on the asset price at the time of prediction, it is necessary to retrieve the previously compiled list of causal variables again, including the observations of the asset in terms of macroeconomic data, industry competition conditions and liquidity levels, and compare these causal variables one by one. According to the strong correlation between the historical time series of each causal variable and price fluctuations, its influence weight in risk prediction is quantified. For example, if a macroeconomic indicator is found to have a high correlation with daily prices, an empirical threshold can be set to mark whether this indicator is at an abnormal level. The specific value of the threshold can come from historical statistics for five years or longer. The maximum and minimum values ​​of the indicator in the long period are combined with the frequency of occurrence for weighted calculation. If it occurs If the increase or decrease of an observation value relative to the empirical threshold exceeds 2%, it is recorded as an important observation point. After comparing the values ​​of multiple important observation points, a relatively independent set of causal influences can be formed, which can be cross-checked with the price at the current prediction point. These cross-check results are then coupled with the price and high, medium and low risk factors are recorded under different circumstances. If certain indicators frequently cross the pre-set high-level range within a certain period of time, the asset is included in the high-level monitoring object. After all these data are written into the asset price and risk prediction model, price fluctuations can be further monitored centrally in subsequent periods and the causal source can be traced back at any time to obtain a completed asset price and risk prediction model.

[0157] The present invention provides a data analysis system, comprising:

[0158] The data collection module organizes the asset prices, market transaction volumes and policy change information of the financial market to obtain an integrated data set;

[0159] The association analysis module uses the integrated data set to analyze the potential relationships between variables, construct a causal graph, and obtain a causal relationship diagram;

[0160] The purification analysis module uses the causal relationship diagram to analyze and eliminate the influence of confounding factors, and applies the double difference model to obtain the purified causal effect results;

[0161] The instrumental variable analysis module uses the causal effect results to select instrumental variables for generalized moment estimation, determine the effectiveness, and obtain the results of the instrumental variable effectiveness analysis;

[0162] The policy impact assessment module uses time series models to predict asset price trends based on the results of instrumental variable effectiveness analysis, builds price and risk prediction models, and obtains dynamic policy evaluation results.

[0163] The above are only preferred embodiments of the present invention and are not intended to limit the present invention in other forms. Any technician familiar with the profession may use the technical contents disclosed above to change or modify them into equivalent embodiments with equivalent changes and apply them to other fields. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A data analysis method based on artificial intelligence, characterized in that: The following steps are involved: Collect and organize variable data of the financial market, including asset prices, market transaction volume and policy change information, to obtain data integration results; based on the data integration results, use the Bayesian network to construct a causal graph, determine the potential relationship between variables, and obtain a causal graph construction result; Based on the result of the causal diagram construction, the causal effect of the target financial variable is derived by applying the Do calculus, and the interference caused by confounding factors is eliminated to obtain the purified causal effect; based on the purified causal effect, the instrumental variables are selected and analyzed, and the validity of the instrumental variables is determined by generalized moment estimation to obtain the instrumental variable analysis results; Using the instrumental variable analysis results, the double difference model is used to eliminate non-policy related noise, quantify the impact of policy changes on the asset market, and obtain the policy impact assessment results; Based on the policy impact assessment results, determine the policy impact in each time window to obtain a dynamic policy assessment result; Based on the dynamic policy evaluation results, the time series model is used to predict asset price trends. At the same time, referring to the causal variable input, an asset price and risk prediction model is constructed.

2. The artificial intelligence-based data analysis method according to claim 1, characterized in that: The steps for obtaining the data integration result are: Collect asset prices, market trading volume and policy change information from financial market databases, which are derived from exchange reports, government announcements and market analysis reports to obtain a preliminary data set; Based on the preliminary data set, perform data cleaning and preprocessing to remove duplicate records, calibrate data format and resolve missing values ​​to obtain a cleaned data set; According to the cleaned data set, the data is integrated through data fusion to obtain a data integration result.

3. The data analysis method based on artificial intelligence according to claim 1, characterized in that: The steps for obtaining the causal graph construction result are: According to the data integration results, nodes and edges are selected to design a Bayesian network, where nodes represent various financial variables, including asset prices and market trading volumes, and edges represent potential causal relationships between variables, thereby obtaining a preliminary causal relationship model; Based on the preliminary causal relationship model, the conditional dependencies between nodes are calculated to obtain the probability distribution. The calculation formula is: Among them, X i represents financial variables, X pa(i) Represents X i The parent node, x i Represents X i The observed value, μ pa(i) and σ pa(i) They are X pa(i) The mean and standard deviation of P(X i ∣X pa(i) ) means that in the parent node X pa(i) Under the condition of X i The probability distribution of According to the probability distribution, the structure of the Bayesian network is updated to obtain a causal graph construction result.

4. The artificial intelligence-based data analysis method according to claim 1, characterized in that: The steps for obtaining the purified causal effect are: According to the result of the cause-effect diagram construction, the target financial variables and influencing factors are determined, and the Do calculation method is used to determine the direct and indirect impact of each factor on the target variable. The causal path between the variables is obtained by constructing an impact chain, and the preliminary analysis results of the causal effect are obtained; Based on the preliminary causal effect analysis results, identify and eliminate the confounding factors that affect the target variable to obtain adjusted causal effect analysis results; The adjusted causal effect analysis results are used to recalculate and evaluate the pure causal relationship of the target financial variable to obtain the purified causal effect.

5. The artificial intelligence-based data analysis method according to claim 1, characterized in that: The steps for obtaining the instrumental variable analysis results are as follows: Based on the purified causal effects, potential instrumental variables associated with the target financial variable are selected; Based on the potential instrumental variables, the relationship strength between each instrumental variable and the target financial variable is evaluated to obtain the estimated parameters. The calculation formula is: Where W represents the instrumental variable matrix, M is the weight matrix derived from the causal graph model, and Y is the observed value of the target variable. represents the estimated parameters; Based on the estimated parameters, the effectiveness of the instrumental variables is evaluated to verify whether the instrumental variables can estimate the causal effect of the target financial variables, and the results of the instrumental variable analysis are obtained.

6. The artificial intelligence-based data analysis method according to claim 1, characterized in that: The steps for obtaining the policy impact assessment results are as follows: Based on the results of the instrumental variable analysis, the difference in market reactions before and after the implementation of the policy is calculated to obtain the policy impact. The calculation formula is: Among them, Y t2 and Y t1 Represent the market performance of the experimental group before and after the policy implementation, C t2 and C t1 Respectively represent the performance of the control group at the same time, N represents the number of observations, and ΔImpact is the policy impact after adjustment for the reference time effect; The policy impact quantity is used to evaluate the policy impact, verify whether the policy change has a statistical impact, determine the performance of the policy impact in the asset market, and obtain the policy impact evaluation result.

7. The artificial intelligence-based data analysis method according to claim 1, characterized in that: The steps for obtaining the dynamic policy evaluation results are as follows: Based on the policy impact assessment results, obtain market performance data within each time window to form an organizational data set sorted by time window; Based on the organizational data set sorted by time windows, identifying the difference in market response before and after the policy change, determining the policy impact of each window by comparing and analyzing the data changes in each time window, and obtaining the time window policy impact analysis results; Based on the policy impact analysis results of the time window, the effect of policy changes on the market is evaluated through continuous time series analysis to obtain dynamic policy evaluation results.

8. The artificial intelligence-based data analysis method according to claim 1, characterized in that: The steps for obtaining the asset price and risk prediction model are as follows: Based on the dynamic policy evaluation results, the ARIMA model is used to predict asset price trends and obtain asset prices at the prediction time. The formula is: P t =α+βP t-1 +∈ t Among them, P t represents the asset price at the prediction time, P t-1 represents the asset price at the previous point in time, α and β are model parameters, ∈ t is the error term; Based on the asset price at the prediction time, the impact of causal variables is integrated to construct an asset price and risk prediction model.

9. A data analysis system according to any one of claims 1 to 8, characterized in that: include: The data collection module organizes the asset prices, market transaction volumes and policy change information of the financial market to obtain an integrated data set; The association analysis module uses the integrated data set to analyze the potential relationships between variables, construct a causal graph, and obtain a causal relationship diagram; The purification analysis module uses the causal relationship diagram to analyze and eliminate the influence of confounding factors, and applies the double difference model to obtain the purified causal effect results; The instrumental variable analysis module uses the causal effect results to select instrumental variables for generalized moment estimation, determine the effectiveness, and obtain the results of the instrumental variable effectiveness analysis; The policy impact assessment module uses time series models to predict asset price trends based on the results of instrumental variable effectiveness analysis, builds price and risk prediction models, and obtains dynamic policy evaluation results.

Images