Financial product data processing system based on multi-source data application

Abstract

The invention relates to the technical field of financial product data processing, in particular to a financial product data processing system based on multi-source data application, which comprises a data collection module, a data weight calculation module, a data label calculation module, a model construction module and a model application module, wherein the data collection module collects data from historical financial product data and real-time financial product data; the data weight calculation module calculates a standardized feature weight based on the moving average line; the data label calculation module generates an abnormal probability label by using a double standard deviation method; the model construction module uses the feature weight and the abnormal probability label to train a regression algorithm model; the model application module predicts a real-time data abnormal probability and calculates a feature importance sequence; the system solves the problems of multi-dimensional data integration, anomaly detection precision, risk factor identification and the like, and realizes multi-source data fusion, dynamic fluctuation analysis, high-precision anomaly detection and interpretable risk analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of financial product data processing, and in particular to a financial product data processing system based on multi-source data application. Background Art

[0002] Traditional financial product data processing systems face many technical challenges when dealing with complex and ever-changing financial market data.

[0003] First, these systems often have difficulty in effectively integrating and utilizing multi-source data, resulting in an incomplete understanding of market conditions. Second, they lack flexibility and accuracy in allocating weights to data features, and are unable to dynamically adapt to market changes. Furthermore, traditional systems often use fixed thresholds or simple statistical methods for anomaly detection, making it difficult to capture subtle market fluctuations and potential risks. In addition, these systems have limited capabilities in real-time data processing and forecasting, making it difficult to provide timely market insights. At the same time, traditional systems generally lack the interpretability of forecast results, making it difficult to provide decision makers with specific risk factor analysis.

[0004] Finally, in terms of model construction and application, traditional systems often have difficulty effectively combining historical data with real-time data, resulting in insufficient accuracy and adaptability of forecasting models. These technical problems combined have severely limited the application effect and decision support capabilities of traditional financial product data processing systems in complex financial environments. Summary of the invention

[0005] The purpose of the present invention is to provide a financial product data processing system based on multi-source data application to solve the problems raised in the above background technology.

[0006] To achieve the above-mentioned purpose, the present invention provides the following technical solution: a financial product data processing system based on multi-source data application, comprising a data collection module, a data weight calculation module, a data label calculation module, a model construction module and a model application module, wherein: The data collection module collects historical financial product data and real-time financial product data from multiple sources; The data weight calculation module calculates the corresponding moving averages according to different data features in the historical financial product data; and calculates the deviation percentages of different data features according to the moving averages of different data features, and standardizes them as the corresponding data feature weights; The data label calculation module extracts the moving average of the price data according to the moving averages of different data features; based on the moving average of the price data and the price data, sets the confidence interval of the price data using the 2 standard deviation method, and uses the deviation percentage of the price data from the confidence interval as the abnormal probability label of the historical financial product data corresponding to the price data; The model building module performs weighted summation of different data features and corresponding data feature weights in the historical financial product data as weight data, and uses the weight data and corresponding abnormal probability labels as training data for the regression algorithm model; The model application module uses the real-time financial product data as the input of the constructed regression algorithm model, predicts the abnormal probability of the real-time financial product data, and compares the moving averages of different data features with the data features in the real-time financial product data to obtain the feature importance of the data features in the real-time financial product data to the predicted abnormal probability, and outputs them to the front end in descending order of feature importance.

[0007] As a further improvement of the present technical solution, the data features in the historical financial product data in the data collection module include product ID, timestamp, price data and transaction volume data.

[0008] As a further improvement of the technical solution, the process of calculating the weights of the data features in the historical financial product data by the data weight calculation module specifically includes: For the numerical data features in the historical financial product data, the corresponding moving average is calculated respectively. The moving average is the average value of the data points within a certain time window. For the numerical data features in the historical financial product data, calculate the historical deviation percentage between the current value and the corresponding moving average. The historical deviation percentage = (current value - moving average) / moving average * 100%; The historical deviation percentages of all numerical data features are standardized so that their sum is 1. The weight of each numerical data feature is obtained by calculating the sum of the absolute values ​​of the historical deviation percentages of all numerical data features and dividing the absolute value of the historical deviation percentage of each numerical data feature by this sum.

[0009] As a further improvement of the technical solution, the data label calculation module calculates the abnormal probability label of the historical financial product data in the following steps: From the moving averages of different numerical data features, extract the moving average of price data as the price moving average; Using the same time window as in the data weight calculation module, calculate the standard deviation between the price data and the price moving average. The standard deviation calculation formula is √(Σ(X-MA)² / N), where X is each price data point, MA is the value of the price moving average, N is the number of data points in the time window, √ is the square root operator, and Σ is the summation operator; The confidence interval of price data is set using the 2-standard deviation method. The upper limit of the confidence interval = price moving average + 2*standard deviation, and the lower limit of the confidence interval = price moving average - 2*standard deviation. For each price data point, calculate its deviation percentage from the confidence interval boundary. If the price is higher than the upper limit of the confidence interval, the deviation percentage = (price data - upper limit of the confidence interval) / upper limit * 100%; if the price is lower than the lower limit of the confidence interval, the deviation percentage = (lower limit of the confidence interval - price data) / lower limit * 100%; if the price is within the confidence interval, the deviation percentage = 0%; The calculated deviation percentage is directly used as the abnormal probability label of the historical financial product data corresponding to the price data point, and the range of the abnormal probability label is 0 to 100%.

[0010] As a further improvement of the technical solution, the training process of the regression algorithm model in the model building module specifically includes: Obtain the numerical data features in the historical financial product data of the data collection module, the weight of each numerical data feature calculated by the data weight calculation module, and the abnormal probability label calculated by the data label calculation module; For the historical financial product data at each time point, multiply the value of each numerical data feature by its corresponding weight, add up all weighted feature values, and obtain the weighted data at that time point; The weight data at each time point is used as the input feature, the abnormal probability label at the corresponding time point is used as the target variable, and the input feature and the target variable are used together as the training data of the regression algorithm model; Regression algorithm models include linear regression, random forest regression, and support vector regression. The regression algorithm model parameters are initialized and the regression model is trained using the prepared regression algorithm model training data. The model parameters are optimized by minimizing the error between the predicted value and the target variable, and finally the regression algorithm model is constructed.

[0011] As a further improvement of the technical solution, the process of the model application module evaluating the feature importance of data features in real-time financial product data to the predicted abnormal probability specifically includes: Acquire real-time financial product data collected by the data collection module, including each numerical data feature, and the weight of each numerical data feature calculated by the data weight calculation module; For real-time financial product data, multiply the value of each numerical data feature by its corresponding weight, add up all weighted feature values, and obtain real-time weighted data; Input the calculated real-time weight data into the regression algorithm model built by the model building module to obtain the abnormal probability predicted by the model; When the predicted abnormal probability is not 0, it means that the data features in the real-time financial product data may be abnormal. Then, the moving average of each numerical data feature in the real-time financial product data is obtained from the data weight calculation module, and its real-time deviation percentage from the moving average is calculated, and the feature importance is calculated. Feature importance = |real-time deviation percentage|*data feature weight; According to the calculated feature importance, all numerical data features are sorted from large to small, and the sorting results and predicted abnormality probabilities are output to the front end.

[0012] Compared with the prior art, the present invention has the following beneficial effects: 1. The financial product data processing system based on multi-source data application realizes comprehensive analysis and anomaly detection of financial product data through multiple applications of moving averages. First, the system calculates the corresponding moving averages according to different data features in the historical financial product data, and uses these moving averages to calculate the deviation percentage of each feature, and then obtains the standardized data feature weights. This process fully considers the historical fluctuations of each feature and lays the foundation for subsequent analysis. Secondly, the system specifically extracts the moving average of price data, combines the 2-times standard deviation method to set the confidence interval, and generates accurate anomaly probability labels by calculating the deviation percentage of price data from the confidence interval. This method not only considers the historical trend of prices, but also can sensitively capture abnormal fluctuations. Finally, the system compares the real-time financial product data with the moving averages of each feature and calculates the feature importance. This step enables the system to quickly identify the factors that have the greatest impact on abnormal prediction. Through these multiple applications of moving averages, the system realizes dynamic analysis, anomaly detection and risk factor identification of financial data, greatly improving the efficiency and accuracy of financial risk management.

[0013] 2. The financial product data processing system based on multi-source data application significantly improves the accuracy and practicality of financial product data analysis through multiple applications of data feature weights. First, the system performs weighted summation of different data features in historical financial product data and their corresponding weights to obtain comprehensive weighted data. This step effectively compresses multi-dimensional feature information into a single value, which not only retains the relative importance of each feature, but also simplifies the subsequent modeling process. Then, the system combines these weighted data with the previously calculated abnormal probability labels as training data for the regression algorithm model. This method ensures that the model can fully consider the importance of each feature during the learning process, thereby improving the accuracy of prediction. In the real-time application stage, the system uses feature weights again to accurately identify the factors that have the greatest impact on abnormal prediction by calculating feature importance (feature importance = |real-time deviation percentage|*weight). This method not only provides abnormal prediction results, but also gives the main reasons for the prediction results, greatly enhancing the interpretability and practical value of the model. Through these multiple applications of data feature weights, the system achieves accurate analysis, effective modeling and clear interpretation of financial data, providing strong support for financial decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic diagram of the overall module of the present invention.

[0015] In the figure: 100, data collection module; 200, data weight calculation module; 300, data label calculation module; 400, model construction module; 500, model application module. DETAILED DESCRIPTION

[0016] The following will be combined with the accompanying drawings in the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] Unless otherwise specified, the terms (including technical terms) used herein have the commonly understood meanings to those skilled in the art. In addition, it is understood that the terms defined in commonly used dictionaries should be understood to have the same meanings as those in the context of the relevant fields, and should not be understood as idealized or overly formal meanings.

[0018] The following are some definitions of terms: Historical financial product data: financial product data over a period of time (such as months, years, or decades) in the past, including features such as product ID, timestamp, price data, and transaction volume data, which are used for model training and statistical indicator calculation; Real-time financial product data: financial product data that is currently occurring or has occurred recently, with the same characteristics as historical data, used for instant decision-making and anomaly detection; Moving averages of different data features: Moving averages calculated for each numerical feature (such as price, trading volume) in historical financial product data, reflecting the average trend of the feature within a certain time window; Data feature weight: The weight value calculated based on the historical deviation percentage of each numerical feature, which is used to measure the impact of different features on the status of financial products; Moving average of price data: specifically refers to the moving average calculated for price data, which is used for subsequent confidence interval calculation and anomaly detection; Confidence interval: The price fluctuation range calculated based on the price moving average and standard deviation. The upper and lower limits are usually set using the 2 times standard deviation method to determine whether the price is abnormal. Abnormal probability label: the percentage of deviation between the price data and the confidence interval, ranging from 0% to 100%, used to indicate the degree of abnormality of historical data points and serves as the target variable for model training; Feature importance: The contribution of each feature in real-time data to the predicted anomaly probability. The calculation method is the absolute value of the real-time deviation percentage of the feature multiplied by its weight, which is used to explain the model prediction results.

[0019] Next, see Figure 1 The present invention provides a technical solution: a financial product data processing system based on multi-source data application, including a data collection module 100, a data weight calculation module 200, a data label calculation module 300, a model construction module 400 and a model application module 500.

[0020] The data collection module 100 collects historical financial product data from multiple sources. The historical financial product data covers data from a period of time in the past, such as data from several months, several years or several decades, and is used to train models, calculate various statistical indicators (such as moving averages, standard deviations), etc. The data features in the historical financial product data include product IDs, timestamps, price data, and transaction volume data, etc.; The data collection module 100 collects real-time financial product data, which is data that is currently occurring or has occurred recently and is used for anomaly detection, making instant trading decisions, etc., and has the same data characteristics as historical financial product data.

[0021] The data weight calculation module 200 calculates the corresponding moving averages according to different data features in the historical financial product data; and calculates the deviation percentages of different data features according to the moving averages of different data features, and performs standardization processing on them, so that the sum of the deviation percentages corresponding to different data features is 1, and serves as the corresponding data feature weights, specifically including: For the numerical data features in the historical financial product data (such as price data, transaction volume data, excluding timestamps), the corresponding moving averages are calculated respectively. The moving average is the average value of the data points in a certain time window. For example, assuming there is price data for the past 20 days, recorded as P1, P2, P3...P20, the data values ​​of the first 20 days are summed up and then divided by 20, that is, (P1+P2+P3+...+P20) / 20. The result is the 20-day moving average value of the 20th day, recorded as MA20; for the 21st day, P1 needs to be removed, the new P21 needs to be added, and then MA21 is calculated, that is, MA21=(P2+P3+...+P20+P21) / 20; For the numerical data features in the historical financial product data, calculate the historical deviation percentage between the current value and the corresponding moving average. The historical deviation percentage = (current value - moving average) / moving average * 100%; The historical deviation percentages of all numerical data features are standardized so that their sum is 1. The weight of each numerical data feature is obtained by calculating the sum of the absolute values ​​of the historical deviation percentages of all numerical data features and dividing the absolute value of the historical deviation percentage of each numerical data feature by this sum.

[0022] The data label calculation module 300 extracts the moving average of the price data according to the moving averages of different data features; based on the moving average of the price data and the price data, sets the confidence interval of the price data using the 2 standard deviation method, and uses the deviation percentage of the price data from the confidence interval as the abnormal probability label of the historical financial product data corresponding to the price data, specifically including: From the moving averages of different numerical data features, extract the moving average of price data as the price moving average; Using the same time window as in the data weight calculation module 200, the standard deviation between the price data and the price moving average is calculated. The standard deviation calculation formula is √(Σ(X-MA)² / N), where X is each price data point, MA is the value of the price moving average, N is the number of data points in the time window, √ is a square root operator, and Σ is a summation operator; The confidence interval of price data is set using the 2-standard deviation method. The upper limit of the confidence interval = price moving average + 2*standard deviation, and the lower limit of the confidence interval = price moving average - 2*standard deviation. For each price data point, calculate its deviation percentage from the confidence interval boundary. If the price is higher than the upper limit of the confidence interval, the deviation percentage = (price data - upper limit of the confidence interval) / upper limit * 100%; if the price is lower than the lower limit of the confidence interval, the deviation percentage = (lower limit of the confidence interval - price data) / lower limit * 100%; if the price is within the confidence interval, the deviation percentage = 0%; The calculated deviation percentage is directly used as the abnormal probability label of the historical financial product data corresponding to the price data point. The range of the abnormal probability label is 0 to 100%, where 0% means completely normal and the closer to 100% means the more likely it is abnormal.

[0023] The model building module 400 performs weighted summation of different data features and corresponding data feature weights in the historical financial product data as weight data, and uses the weight data and corresponding abnormal probability labels as training data for the regression algorithm model, specifically including: Obtaining numerical data features (such as price data, transaction volume data, etc.) in historical financial product data of the data collection module 100, as well as the weight of each numerical data feature calculated by the data weight calculation module 200, and the abnormal probability label calculated by the data label calculation module 300; For the historical financial product data at each time point, multiply the value of each numerical data feature by its corresponding weight, add up all weighted feature values, and obtain the weighted data at that time point; The weight data at each time point is used as the input feature, the abnormal probability label at the corresponding time point is used as the target variable, and the input feature and the target variable are used together as the training data of the regression algorithm model; Regression algorithm models include linear regression, random forest regression, and support vector regression. The regression algorithm model parameters are initialized and the regression model is trained using the prepared regression algorithm model training data. The model parameters are optimized by minimizing the error between the predicted value and the target variable (abnormal probability label), and finally the regression algorithm model is constructed.

[0024] The model application module 500 uses the real-time financial product data as the input of the constructed regression algorithm model, predicts the abnormal probability of the real-time financial product data, and compares the moving average of different data features with the data features in the real-time financial product data to obtain the feature importance of the data features in the real-time financial product data to the predicted abnormal probability, and outputs them to the front end in descending order of feature importance, specifically including: Acquire the real-time financial product data collected by the data collection module 100, including various numerical data features (such as price data, transaction volume data, etc.), and the weight of each numerical data feature calculated by the data weight calculation module 200; For real-time financial product data, multiply the value of each numerical data feature by its corresponding weight, add up all weighted feature values, and obtain real-time weighted data; Input the calculated real-time weight data into the regression algorithm model constructed by the model construction module 400 to obtain the abnormal probability predicted by the model; When the predicted abnormal probability is not 0, it means that the data features in the real-time financial product data may be abnormal, then the moving average of each numerical data feature in the real-time financial product data is obtained from the data weight calculation module 200, and the real-time deviation percentage from the moving average is calculated, and the feature importance is calculated, feature importance = |real-time deviation percentage|*data feature weight; According to the calculated feature importance, all numerical data features are sorted from large to small, and the sorting results and predicted abnormality probabilities are output to the front end.

[0025] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A financial product data processing system based on multi-source data application, characterized in that: The system comprises a data collection module (100), a data weight calculation module (200), a data label calculation module (300), a model construction module (400) and a model application module (500), wherein: The data collection module (100) collects historical financial product data and real-time financial product data from multiple sources; The data weight calculation module (200) calculates corresponding moving averages according to different data features in the historical financial product data; and calculates the deviation percentages of different data features according to the moving averages of different data features, and performs standardization processing on them as corresponding data feature weights; The data label calculation module (300) extracts a moving average of price data based on moving averages of different data features; based on the moving average of price data and price data, sets a confidence interval of price data using a 2-times standard deviation method, and uses the deviation percentage of price data from the confidence interval as an abnormal probability label of historical financial product data corresponding to the price data; The model building module (400) performs weighted summation of different data features and corresponding data feature weights in the historical financial product data as weight data, and uses the weight data and corresponding abnormal probability labels as training data for the regression algorithm model; The model application module (500) uses the real-time financial product data as input to the constructed regression algorithm model, predicts the abnormal probability of the real-time financial product data, and compares the moving averages of different data features with the data features in the real-time financial product data to obtain the feature importance of the data features in the real-time financial product data to the predicted abnormal probability, and outputs them to the front end in descending order of feature importance.

2. The financial product data processing system based on multi-source data application according to claim 1, characterized in that: The data features in the historical financial product data in the data collection module (100) include product ID, timestamp, price data and transaction volume data.

3. The financial product data processing system based on multi-source data application according to claim 1, characterized in that: The process of calculating the weight of the data features in the historical financial product data by the data weight calculation module (200) specifically includes: For the numerical data features in the historical financial product data, the corresponding moving average is calculated respectively. The moving average is the average value of the data points within a certain time window. For the numerical data features in the historical financial product data, calculate the historical deviation percentage between the current value and the corresponding moving average. The historical deviation percentage = (current value - moving average) / moving average * 100%; The historical deviation percentages of all numerical data features are standardized so that their sum is 1. The weight of each numerical data feature is obtained by calculating the sum of the absolute values ​​of the historical deviation percentages of all numerical data features and dividing the absolute value of the historical deviation percentage of each numerical data feature by this sum.

4. The financial product data processing system based on multi-source data application according to claim 3, characterized in that: The data label calculation module (300) calculates the abnormal probability label of the historical financial product data in the following steps: From the moving averages of different numerical data features, extract the moving average of price data as the price moving average; Using the same time window as in the data weight calculation module (200), calculate the standard deviation between the price data and the price moving average. The standard deviation calculation formula is √(Σ(X-MA)² / N), where X is each price data point, MA is the value of the price moving average, N is the number of data points in the time window, √ is the square root operator, and Σ is the summation operator; The confidence interval of price data is set using the 2-standard deviation method. The upper limit of the confidence interval = price moving average + 2*standard deviation, and the lower limit of the confidence interval = price moving average - 2*standard deviation. For each price data point, calculate its deviation percentage from the confidence interval boundary. If the price is higher than the upper limit of the confidence interval, the deviation percentage = (price data - upper limit of the confidence interval) / upper limit * 100%; if the price is lower than the lower limit of the confidence interval, the deviation percentage = (lower limit of the confidence interval - price data) / lower limit * 100%; if the price is within the confidence interval, the deviation percentage = 0%; The calculated deviation percentage is directly used as the abnormal probability label of the historical financial product data corresponding to the price data point, and the range of the abnormal probability label is 0 to 100%.

5. The financial product data processing system based on multi-source data application according to claim 4, characterized in that: The training process of the regression algorithm model in the model building module (400) specifically includes: Acquiring numerical data features in historical financial product data from a data collection module (100), the weight of each numerical data feature calculated by a data weight calculation module (200), and an abnormal probability label calculated by a data label calculation module (300); For the historical financial product data at each time point, multiply the value of each numerical data feature by its corresponding weight, add up all weighted feature values, and obtain the weighted data at that time point; The weight data at each time point is used as the input feature, the abnormal probability label at the corresponding time point is used as the target variable, and the input feature and the target variable are used together as the training data of the regression algorithm model; Regression algorithm models include linear regression, random forest regression, and support vector regression. The regression algorithm model parameters are initialized and the regression model is trained using the prepared regression algorithm model training data. The model parameters are optimized by minimizing the error between the predicted value and the target variable, and finally the regression algorithm model is constructed.

6. The financial product data processing system based on multi-source data application according to claim 5, characterized in that: The process of the model application module (500) evaluating the feature importance of data features in real-time financial product data to predicted abnormal probability specifically includes: Acquire real-time financial product data collected by the data collection module (100), including each numerical data feature, and the weight of each numerical data feature calculated by the data weight calculation module (200); For real-time financial product data, multiply the value of each numerical data feature by its corresponding weight, add up all weighted feature values, and obtain real-time weighted data; Inputting the calculated real-time weight data into the regression algorithm model constructed by the model construction module (400) to obtain the abnormal probability predicted by the model; When the predicted abnormal probability is not 0, it indicates that the data feature in the real-time financial product data may be abnormal, then the moving average of each numerical data feature in the real-time financial product data is obtained from the data weight calculation module (200), the real-time deviation percentage from the moving average is calculated, and the feature importance is calculated, where feature importance = |real-time deviation percentage|*data feature weight; According to the calculated feature importance, all numerical data features are sorted from large to small, and the sorting results and predicted abnormality probabilities are output to the front end.

Images