A non-real-time data monitoring and early warning method and system for a third-party data information platform

Abstract

The present invention discloses a non-real-time data monitoring and early warning method and system for a third-party data information platform, including: collecting data source scores of a third-party data information platform for several data sources; preprocessing the data source scores to obtain processing data source scores; calculating the change rate of the real-time technical indicators of the processing data source scores compared to the historical technical indicators of the processing data source scores, and monitoring and early warning the change rate; when determining that the change rate is abnormal, determining the data products corresponding to the abnormal change rate, generating an abnormal product list; visualizing the abnormal product list. The non-real-time data monitoring and early warning method provided by the present invention can enable business personnel to promptly and efficiently determine the anomalies in the data source call process, quickly troubleshoot problems, ensure the normal operation of data source products and supporting systems, and avoid economic losses.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a non-real-time data monitoring and early warning method and system for a third-party data information platform. Background Art

[0002] In the era of big data, data resources are key data for enterprise production. Especially in the field of financial credit, abnormalities in data resources can lead to errors or paralysis of the production system, causing irreparable economic losses. Therefore, it is very necessary to monitor and warn data resources. As a data call agency of the third-party data information platform, it is necessary to establish a complete monitoring and early warning mechanism to predict the data source before the abnormality occurs, take precautions before it happens, and prevent errors in the production system caused by delayed response. Moreover, after the abnormality occurs in the data source, the data call agency should quickly locate the fault of the data source and make a response plan to reduce the economic losses caused by the abnormality of the data source to production. At present, most of the data source monitoring and early warning systems on the market only monitor the real-time technical indicators of the data source, but ignore the time correlation of the change rate of the real-time technical indicators, and lack the depth of monitoring. A small number of more advanced data source monitoring and early warning systems have the ability to monitor and warn the data source based on the time correlation of the change rate of the real-time technical indicators, but this monitoring and early warning system has the defect of being unable to predict the technical indicators of the data source, and still needs to be improved. Summary of the invention

[0003] The present invention aims to solve at least one of the technical problems in the above-mentioned technology to a certain extent. To this end, the purpose of the present invention is to propose a non-real-time data monitoring and early warning method and system for a third-party data information platform, by determining the change trend of the change rate of the technical indicators of the data source score according to the third-party data information platform's score of the data source, and monitoring and early warning the data source by predicting the change trend.

[0004] To achieve the above-mentioned purpose, the first embodiment of the present invention proposes a non-real-time data monitoring and early warning method for a third-party data information platform, comprising:

[0005] Collect data source scores of several data sources from third-party data information platforms;

[0006] Preprocessing the data source score to obtain a processed data source score;

[0007] Calculate the change rate of the real-time technical indicator of the processing data source score compared to the historical technical indicator of the processing data source score, and monitor and warn the change rate;

[0008] When determining that the change rate is abnormal, determining the data product corresponding to the change rate abnormality, and generating a list of abnormal products;

[0009] The abnormal product list is visualized.

[0010] To achieve the above-mentioned purpose, the second embodiment of the present invention proposes a non-real-time data monitoring and early warning system for a third-party data information platform, including:

[0011] The data source submodule is used to collect data source scores of several data sources from third-party data information platforms;

[0012] A data preprocessing submodule, used to preprocess the data source score to obtain a processed data source score;

[0013] A monitoring technical indicator calculation submodule is used to calculate the change rate of the real-time technical indicator of the processing data source score compared with the historical technical indicator of the processing data source score, and monitor and warn the change rate;

[0014] The early warning rule submodule is used to determine the data product corresponding to the abnormal change rate when the change rate is determined to be abnormal, and generate a list of abnormal products;

[0015] The early warning report display submodule is used to visualize the list of abnormal products.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] 1. Monitor the data source based on the rate of change of the technical indicators of the data source score over time. To a certain extent, the relationship between the rate of change of the technical indicators of the data source score and time can be determined, which is convenient for re-analyzing the changes in the data source;

[0018] 2. The present invention provides a method for predicting technical indicators of data sources. Through this method, the changing trend of the technical indicators of data source scoring can be predicted, and an early warning can be issued before the data source scoring may be abnormal, thereby improving the troubleshooting speed and reducing losses.

[0019] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0020] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0022] Figure 1 It is a schematic diagram of a non-real-time data monitoring and early warning method of a third-party data information platform according to an embodiment of the present invention;

[0023] Figure 2 It is a block diagram of a non-real-time data monitoring and early warning system of a third-party data information platform according to an embodiment of the present invention;

[0024] Figure 3 4 is a block diagram of a change rate prediction monitoring submodule according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0026] like Figure 1 As shown, the present invention provides a non-real-time data monitoring and early warning method for a third-party data information platform, which is characterized by comprising S1-S5:

[0027] S1. Collect data source scores of several data sources from third-party data information platforms;

[0028] S2. Preprocessing the data source score to obtain a processed data source score;

[0029] S3. Calculate the change rate of the real-time technical indicators of the processing data source score compared to the historical technical indicators of the processing data source score, and monitor and warn the change rate;

[0030] S4. When determining that the change rate is abnormal, determining the data product corresponding to the abnormal change rate, and generating a list of abnormal products;

[0031] S5. Visualize the list of abnormal products.

[0032] The working principle of the above technical solution is as follows: the third-party data information platform is the caller of data products, data materials or data sources. After obtaining big data products or big data materials on the third-party data information platform, the third-party data information platform collects feedback information on the data source of the big data materials, and screens the scores of the third-party data information platform on several data sources from the feedback information, pre-processes the data source scores, and obtains the processed data source scores. The real-time technical indicators of the data source score are calculated, including the total value of the data source score calculated by summing up the score values ​​of several data sources, the total value of the normal data source score calculated by summing up the score values ​​of several data sources, and the effective call rate. After the real-time technical indicators are calculated, the historical technical indicators are obtained from the historical information of the data source score, and the change rate of the real-time technical indicators compared with the historical technical indicators is calculated and monitored. When the change rate is too high, too low, or the mutation speed is fast, the change rate is determined to be abnormal, and the data products corresponding to the abnormal change rate are counted to generate a list of abnormal products, and the list of abnormal products is visualized and converted into easy-to-read graphic information.

[0033] Beneficial effects of the above technical solutions: The third-party data information platform is the caller of the big data product. Compared with scoring the data source by other means, the third-party data information platform's scoring of the data source can better reflect the value of the data source in practical applications. Scoring the data source based on its scoring of the data source can ensure that the evaluation of the data source is more objective and fair. After obtaining the data source score, the data source score is preprocessed. During the preprocessing process, a part of the abnormal data source score can be screened out, reflecting some problems of the data source, which is convenient for timely early warning of the data source. For details, please refer to the following. According to the rate of change of the technical indicators of the data source score over time, the data source is monitored. To a certain extent, the relationship between the rate of change of the technical indicators of the data source and time can be determined, which is convenient for re-analyzing the changes in the data source. After determining that the data source is abnormal, a list of abnormal data sources is generated, so that business personnel can correct the problems of the data source in a targeted manner. Visualizing the list of abnormal data sources further enables business personnel to quickly troubleshoot the problems of abnormal data sources, ensuring the normal operation of the system and avoiding economic losses.

[0034] According to some embodiments of the present invention, preprocessing the data source score to obtain a processed data source score includes:

[0035] Obtaining a plurality of data source score values ​​corresponding to the plurality of data source scores, each data source score corresponding to a data source score value;

[0036] Performing numerical analysis on the score values ​​of the plurality of data sources to determine a plurality of abnormal score values ​​and a plurality of normal score values ​​among the plurality of score values;

[0037] The data source score corresponding to the normal score value is used as the normal data source score;

[0038] The data source score corresponding to the abnormal score value is used as the abnormal data source score;

[0039] Determining the abnormal type of the abnormal data source score according to the abnormal score value;

[0040] The abnormal score value is replaced with a preset score value, and the replaced abnormal data source score is used as the modified data source score;

[0041] The processed data source score is obtained according to the modified data source score and the normal data source score.

[0042] The working principle of the above technical solution: The third-party data information platform will score each data source, perform numerical analysis on the score value of each data source, and distinguish between normal score values ​​and abnormal score values. For example, when the third-party data information platform scores the data source on a five-point scale, and the score value of one of the data sources is "-35565", obviously, we can determine that the score value of the data source score is an abnormal score value. When the third-party data information platform scores the data source as "Error_Code = -2456475", obviously, the score value of the data source score is also an abnormal score value. Furthermore, we can also determine the error code corresponding to the abnormal score value, and then determine the abnormal type and cause of the score value. After obtaining the abnormal data source score, all abnormal score values ​​are replaced with preset values ​​to obtain the corrected data source score. According to the above preprocessing rules, the data is preprocessed, and the normal data source score and the corrected data source score are used as the processed data source score.

[0043] The beneficial effects of the above technical solution are: numerical analysis is performed on the data source scores, and the values ​​are divided into abnormal score values ​​and normal score values, which is convenient for data correction in the next step. According to the characteristics of the abnormal score value, its abnormal type is determined to facilitate the investigation and resolution of the problem of the third-party data information platform or data source. After determining the abnormal score value, it is filled with the preset value to ensure the capacity of the data source score sample, which is convenient for the next call.

[0044] According to some embodiments of the present invention, calculating the change rate of the real-time technical indicator of the processing data source score compared to the historical technical indicator of the processing data source score includes:

[0045] The number of the normal data source scores is used as the effective call volume of the data source corresponding to the normal data source score by the third-party data information platform;

[0046] The number of processed data source scores is used as the total number of data source calls corresponding to the processed data source scores by the third-party data information platform;

[0047] Calculate the ratio of the effective call volume to the total call volume as the effective call rate;

[0048] The real-time technical indicator is formed according to the plurality of normal scoring values, the plurality of data source scoring values ​​and the effective call rate;

[0049] Obtaining historical technical indicators for processing data source scoring; the historical technical indicators include a number of historical normal scoring values, a number of historical data source scoring values, and a historical effective call rate;

[0050] Determine a first difference value according to the plurality of normal scoring values ​​and the plurality of historical normal scoring values, and determine a first change rate according to the first difference value and the plurality of historical normal scoring values;

[0051] Determine a second difference value according to the score values ​​of the plurality of data sources and the score values ​​of the plurality of historical data sources, and determine a second change rate according to the second difference value and the score values ​​of the plurality of historical data sources;

[0052] Calculating a third difference between the effective call rate and the historical effective call rate, and taking a ratio of the third difference to the historical effective call rate as a third change rate;

[0053] According to the first change rate, the second change rate and the third change rate, a change rate of the real-time technical indicator of the processing data source score compared to the historical technical indicator of the processing data source score is obtained.

[0054] The working principle of the above technical solution is: sum up the normal score values ​​corresponding to the normal data source score to obtain the total normal score value; the total normal score value is one of the technical indicators of the data source score. After obtaining the real-time total normal score value, obtain the historical data of the total normal score value, that is, calculate the total historical normal score value based on the sum of several historical normal score values. The difference between the real-time total normal score value and the historical total normal score value is calculated as the first difference, and then the ratio of the first difference to the historical total normal score value is used as the first change rate. According to the same principle, calculate the change rate of the remaining technical indicators of the data source score (calculate the total data source score value and the effective call rate based on the sum of several data source score values).

[0055] The beneficial effects of the above technical solution are as follows: by analyzing the total normal score of the data source score, the overall status of the data source corresponding to the normal score data source in the process of interacting with the third-party data information platform can be determined. Similarly, by analyzing the total value of the data source score, the overall status of all data sources in the process of interacting with the third-party data information platform can be analyzed. By analyzing the effective call rate, the proportion of normal data source scores in all data source scores can be determined. By analyzing the change rate of real-time technical indicators and historical technical indicators, the trend of data source scores changing over time can be determined.

[0056] According to some embodiments of the present invention, monitoring and warning the change rate includes:

[0057] Acquire first historical data of a change rate of a real-time technical indicator of a score of a data source to be analyzed compared to a historical technical indicator of a score of the data source to be analyzed;

[0058] Setting a sampling period to sample the first historical data of the change rate, obtaining a plurality of change rate sampling values, and establishing a change rate time series of the change rate sampling values ​​with respect to time;

[0059] Determining a confidence interval for the rate of change sample values ​​in the rate of change time series;

[0060] Marking a number of sampling points in the change rate time series that are not within the confidence interval as suspicious sampling points, taking the time point at which each suspicious sampling point is located as a suspicious time point, taking a number of non-suspicious time points adjacent to each suspicious time point as verification time points, and taking the sampling point corresponding to each verification time point as a verification sampling point;

[0061] Obtain the suspicious sampling value corresponding to each suspicious sampling point;

[0062] Obtain the verification sampling value of the verification sampling point corresponding to each suspicious sampling point;

[0063] Calculate the outlier index of the suspicious sampling point according to the suspicious sampling value and a number of verification sampling values ​​corresponding to the suspicious sampling value, and compare it with the outlier index threshold;

[0064] When it is determined that the outlier index is greater than the outlier index threshold, the suspicious sampling point is marked as an abnormal sampling point, and the suspicious time point corresponding to the abnormal sampling point is marked as an abnormal time point, and a change rate time series containing abnormal marks is obtained as an abnormal change rate time series;

[0065] Establish a prediction model for abnormal change rate time series based on neural network technology;

[0066] Using the abnormal change rate time series as training data to train the prediction model, to obtain a target prediction model;

[0067] Among several data source scores, one data source score is randomly selected as the data source score to be analyzed;

[0068] The historical data of the change rate of the real-time technical indicator of the score of the data source to be analyzed compared with the historical technical indicator of the score of the data source to be analyzed is obtained, and input into the target prediction model for monitoring and early warning.

[0069] The calculating the outlier index of the suspicious sampling point according to the suspicious sampling value and a plurality of verification sampling values ​​corresponding to the suspicious sampling value comprises:

[0070] Establish a rectangular coordinate system to obtain the coordinate information of the suspicious sampling points and the verification sampling points;

[0071] Substituting the coordinate information of the suspicious sampling point and a number of verification sampling points corresponding to the suspicious sampling point into an outlier index formula to obtain an outlier index of the suspicious sampling point;

[0072] The outlier index formula is:

[0073]

[0074] Wherein, E is the outlier index of the suspicious sampling point; n is the number of verification sampling points corresponding to the suspicious sampling point; x is the horizontal coordinate of the suspicious sampling point; y is the vertical coordinate of the suspicious sampling point; x k is the horizontal coordinate of the kth verification sampling point corresponding to the suspicious sampling point; k is the ordinate of the kth verification sampling point corresponding to the suspicious sampling point; max(x k ) is the maximum value among the abscissas of the plurality of verification sampling points; min(x k ) is the minimum value among the abscissas of the plurality of verification sampling points; max(y k ) is the maximum value among the vertical coordinates of the plurality of verification sampling points; min(y k ) is the minimum value among the vertical coordinates of the several verification sampling points.

[0075] The working principle of the above technical solution is: collect and sample the first historical data of the rate of change of the technical indicator, obtain several sampling values ​​of the rate of change, and establish a time series of the rate of change in combination with the time points corresponding to the sampling values ​​of the rate of change; determine the confidence interval of the sampling value of the rate of change according to the principle of statistics, mark the sampling points that are not in the interval as suspicious sampling points, and take the time points corresponding to the suspicious sampling points as suspicious time points. Take several non-suspicious time points near the suspicious time points as verification time points. For example, among time points 1 to 10, the suspicious time points are time points 4 and 5. Assuming that we only need to take 6 verification time points before and after the suspicious time points, then the verification time points corresponding to the suspicious time point 4 are time points 1-3 before time point 4 and time points 6-8 after time point 4 (time point 5 is an abnormal time point, so it is skipped when counting the verification time points); establish a rectangular coordinate system, place the suspicious sampling points and their corresponding verification sampling points into the rectangular coordinate system, and calculate the outlier index of the suspicious sampling points according to their coordinate values. The physical meaning of the outlier index formula is: the ratio of the average value of the distance between the suspicious sampling point and each corresponding verification sampling point to the area of ​​the distribution range of the sampling points in the time series; in one embodiment of the present invention, a time series of the rate of change with a sampling value confidence interval of (6.21, 7.55) is:

[0076] Time 11 12 13 14 15 16 Sample value 7.32 6.32 7.37 7.20 6.46 9.31 Time 17 18 19 20 21 …… Sample value 7.49 6.90 6.80 7.53 6.94 ……

[0077] Obviously, in the change rate time series, the suspicious sampling point is the sampling point at time point 16, and the corresponding sampling value is 9.31. The point corresponding to time point 16 is taken as the suspicious sampling point, and the sampling points corresponding to the time points other than time point 16 in the sequence are taken as the verification sampling points. The above outlier index formula is input, and its outlier index is 0.322; in this embodiment, the outlier index threshold is 0.25. Obviously, the suspicious sampling point should be recorded as an abnormal sampling point. In this embodiment, it is assumed that when the suspicious sampling value changes to 9.0, its outlier index is 0.255, which obviously exceeds the outlier index threshold, and when the suspicious sampling value changes to 7.55, its outlier index is 0.213, which is obviously lower than the outlier index threshold. Through the above embodiments and two hypothetical examples, it is not difficult to summarize the following rules: the farther the suspicious sampling point deviates from the confidence interval, the higher its outlier index. This theory is consistent with the actual situation. After obtaining the outlier index, it is compared with the preset outlier index threshold. When the outlier index is greater than the threshold, the suspicious sampling point is identified as an abnormal sampling point, marked, and the abnormal change rate time series containing the abnormal mark is obtained. Then, the prediction model is constructed using neural network technology, and the abnormal change rate time series is used as training data to train the prediction model to obtain a target prediction model that can predict the change rate time series. After that, at least one data source score is selected from several data source scores as the data source score to be analyzed, and the data corresponding to the change rate of the technical indicators of the data source score to be analyzed is obtained and input into the target prediction model. The change trend of the change rate of its real-time technical indicators compared with the change rate of the historical technical indicators is predicted, and monitoring and early warning are carried out according to the predicted change trend.

[0078] The beneficial effects of the above technical solutions are as follows: Analyze the historical data of the change rate from the perspective of time to obtain the corresponding relationship between the change rate and time, which is conducive to re-analyzing the change of the change rate, further helping business personnel to control the trend of the change rate and determine the abnormal time point of the change rate; divide the sampling points according to whether they are within the confidence interval, so that most of the sampling points fall within the confidence interval, and only a few sampling points are outside the confidence interval, ensuring that the number of suspicious sampling points that need to calculate the outlier index is small, reducing the amount of calculation. The outlier index, an indicator that combines the time point factor and the sampling value factor, is used to determine whether a suspicious sampling point is an abnormal sampling point, which overcomes the inherent defects of the traditional abnormal analysis of suspicious sampling points based only on a single indicator such as the time point or the sampling value, making the reference significance of the judgment result higher; analyze the change rate of the technical indicators of the data source scoring to be analyzed according to the prediction model trained based on the neural network technology, which has a high degree of automation and can save human resources.

[0079] According to some embodiments of the present invention, obtaining historical data of the change rate of the real-time technical indicator of the score of the data source to be analyzed compared with the historical technical indicator of the score of the data source to be analyzed, and inputting it into the target prediction model for monitoring and early warning includes:

[0080] Acquire second historical data of a change rate of the real-time technical indicator of the score of the data source to be analyzed compared with the historical technical indicator of the score of the data source to be analyzed;

[0081] Sampling the second historical data to obtain a second historical data sequence;

[0082] Sampling the preceding data of the second historical data to obtain a preceding data sequence;

[0083] Inputting the preceding data sequence into the target prediction model for data analysis and prediction to obtain a first prediction data sequence including a plurality of prediction data;

[0084] Calculate the first correlation coefficient between the second historical data sequence and the first predicted data sequence according to the correlation coefficient calculation formula, and compare the first correlation coefficient with a preset first correlation coefficient threshold;

[0085] When it is determined that the first correlation coefficient is greater than the first correlation coefficient threshold, the second historical data sequence is input into the target prediction model to obtain a second prediction data sequence; the first prediction data sequence is input into the target prediction model to obtain a third prediction data sequence;

[0086] Calculating a second correlation coefficient between the second prediction data series and the third prediction data series according to a correlation coefficient calculation formula, and comparing the second correlation coefficient with a preset second correlation coefficient threshold;

[0087] When it is determined that the second correlation coefficient is greater than the second correlation coefficient threshold, performing an abnormality analysis on the second predicted data sequence to obtain predicted suspicious data points and corresponding predicted outlier indexes in the second predicted data sequence, and comparing the predicted outlier index with a preset predicted outlier index threshold;

[0088] When it is determined that the predicted outlier index is greater than a preset predicted outlier index threshold, it is determined that the change trend of the real-time technical indicator of the data source score to be analyzed compared with the change trend of the historical technical indicator of the data source score to be analyzed is abnormal, and an early warning is issued.

[0089] The working principle of the above technical solution is: obtain the second historical data of the technical indicators of the score of the data source to be analyzed. The second historical data is a plurality of continuous data of the technical indicators of the score of the data source to be analyzed within a certain time period; for example, the values ​​corresponding to the technical indicators of the score of the data source to be analyzed from January 2 to January 3 are combined to obtain the second historical data of the score of the data source to be analyzed from January 2 to January 3; the predecessor data of the second historical data is the data of the technical indicators of the score of the data source to be analyzed in the time period before the second historical data; for example, the historical data from January 1 to January 2 can be used as the predecessor data of the second historical data from January 2 to January 3; the second historical data and the predecessor data are sampled respectively to obtain a second historical data sequence and a predecessor data sequence; the predecessor data After the sequence is input into the target prediction model, the predicted data sequence of the previous data sequence can be obtained. For example, after the data sequence corresponding to the historical data from January 1 to January 2 is input into the prediction model, the predicted data sequence for the data from January 2 to January 3 can be obtained, which is used as the first predicted data sequence; then, according to the corresponding relationship of time (for example, the sampling value at 12:00 on January 1 in the previous data sequence corresponds to the sampling value at 12:00 on January 2 in the second historical data sequence), after the corresponding relationship is established, the sampling values ​​of the two sequences are used as variables X and variable Y respectively, and the correlation between variables X and variable Y is calculated according to the Pearson correlation coefficient formula. The Pearson correlation coefficient calculation formula is:

[0090]

[0091] In the above formula, ρ X,Y is the correlation coefficient between variables X and Y, cov(X,Y) represents the covariance between variables X and Y, σ X represents the standard deviation of variable X, σ YRepresents the standard deviation of variable Y, and E is the mathematical expectation. According to the above formula, the first correlation coefficient between the first predicted data sequence and the preceding data sequence is calculated, and compared with the preset first correlation coefficient threshold. When it is determined that the first correlation coefficient is greater than the first correlation coefficient threshold, it means that the target prediction model is more accurate in predicting the future trend of the preceding data sequence. This step belongs to the verification of the performance of the prediction model; then, the second historical data sequence is input into the target prediction model to predict the trend of the second historical data sequence and obtain the second predicted data sequence; at the same time, the first predicted data sequence is input into the target prediction model for prediction to obtain the third predicted data sequence; according to the same method, the second correlation coefficient between the second predicted data sequence and the third predicted data sequence is calculated, and compared with the preset second correlation coefficient threshold. This step belongs to the secondary verification of the prediction model. When it is determined that the second correlation coefficient is greater than the second correlation coefficient threshold, it means that the correlation between the second predicted data sequence and the third predicted data sequence is high, and the target prediction model predicts the trend of the historical data sequence more accurately; then, according to the anomaly analysis described above, the predicted outlier index of the predicted abnormal data point in the second predicted data sequence is calculated, and compared with the preset predicted outlier index threshold. When it is determined that the predicted outlier index is greater than the predicted outlier index threshold, it means that there is a possibility of anomalies in the second historical data sequence in the future, that is, there is an abnormality in the trend of the change rate of the technical indicator of the data source score to be analyzed, and the second historical data sequence is warned. It should be noted that the first correlation coefficient threshold and the second correlation coefficient threshold are determined by multiple experiments. In terms of numerical relationship, the second correlation coefficient threshold is less than or equal to the first correlation coefficient threshold.

[0092] The beneficial effects of the above technical solution are as follows: before performing predictive analysis on the second historical data sequence, predicting the preceding data of the second historical data sequence to obtain a first predicted data sequence, calculating the correlation coefficient between the first predicted data sequence and the second historical data sequence, and verifying the data prediction effect of the target prediction model based on the correlation coefficient, thereby ensuring the reliability of the prediction of the second historical data sequence by the target prediction model; after predicting the second historical data sequence to obtain the second predicted data sequence, predicting the first predicted data sequence to obtain a third predicted data sequence, and calculating the second correlation coefficient between the second predicted data sequence and the third predicted data sequence. This step is a secondary verification of the data prediction effect of the target prediction model, thereby further ensuring the reliability of the prediction of the second historical data sequence by the target prediction model, and ensuring that the second predicted data sequence is the future trend of the second historical data sequence.

[0093] like Figure 2 As shown, the present invention provides a non-real-time data monitoring and early warning system for a third-party data information platform, including:

[0094] The data source submodule is used to collect data source scores of several data sources from third-party data information platforms;

[0095] A data preprocessing submodule, used to preprocess the data source score to obtain a processed data source score;

[0096] A monitoring technical indicator calculation submodule is used to calculate the change rate of the real-time technical indicator of the processing data source score compared with the historical technical indicator of the processing data source score, and monitor and warn the change rate;

[0097] The early warning rule submodule is used to determine the data product corresponding to the abnormal change rate when the change rate is determined to be abnormal, and generate a list of abnormal products;

[0098] The early warning report display submodule is used to visualize the list of abnormal products. The working principle of the above technical solution: after the data source submodule obtains the big data product or big data data from the third-party data information platform, it collects the feedback information of the third-party data information platform on the data source of the big data data, and screens the scores of several data sources from the third-party data information platform from the feedback information; then the data source submodule passes the scores of several data sources to the data preprocessing submodule to preprocess the data source scores to obtain the processed data source scores; then the data source submodule passes the processed data source scores to the monitoring technical indicator calculation submodule to calculate the real-time technical indicators of the data source scores, including summing up the scores based on several data source scores, and calculating the total value of the data source scores, based on the sum of several data source scores. , calculate the total value of the normal data source score, the total value of the data source score, and the effective call rate. After calculating the real-time technical indicators, obtain the historical technical indicators in the historical information of the data source score, calculate the change rate of the real-time technical indicators compared with the historical technical indicators, and monitor and warn; when the change rate of the real-time technical indicators of a data source score is too high, too low, or the mutation speed is fast compared with the historical technical indicators, the change rate is determined to be abnormal, and the early warning rule sub-module counts the data products corresponding to the abnormal change rate, generates an abnormal product list, and transmits it to the early warning report display sub-module, which visualizes the abnormal product list and converts it into easy-to-read graphic information.

[0099] Beneficial effects of the above technical solutions: The third-party data information platform is the caller of the big data product. Compared with scoring the data source by other means, the third-party data information platform's scoring of the data source can better reflect the value of the data source in practical applications. Scoring the data source based on its scoring of the data source can ensure that the evaluation of the data source is more objective and fair. After obtaining the data source score, the data source score is preprocessed. During the preprocessing process, a part of the abnormal data source score can be screened out, reflecting some problems of the data source, which is convenient for timely early warning of the data source. For details, please refer to the following. According to the rate of change of the technical indicators of the data source score over time, the data source is monitored. To a certain extent, the relationship between the rate of change of the technical indicators of the data source and time can be determined, which is convenient for re-analyzing the changes in the data source. After determining that the data source is abnormal, a list of abnormal data sources is generated, so that business personnel can correct the problems of the data source in a targeted manner. Visualizing the list of abnormal data sources further enables business personnel to quickly troubleshoot the problems of abnormal data sources, ensuring the normal operation of the system and avoiding economic losses.

[0100] According to some embodiments of the present invention, the data preprocessing module includes:

[0101] A scoring value acquisition submodule, used to acquire a plurality of data source scoring values ​​corresponding to the plurality of data source scores;

[0102] The rating value discrimination submodule is used to:

[0103] Performing numerical analysis on the score values ​​of the plurality of data sources to determine a plurality of abnormal score values ​​and a plurality of normal score values ​​among the plurality of score values;

[0104] The data source score corresponding to the normal score value is used as the normal data source score;

[0105] The data source score corresponding to the abnormal score value is used as the abnormal data source score;

[0106] An anomaly type determination submodule, used to determine the anomaly type of the anomaly data source score according to the anomaly score value;

[0107] An abnormal data source score correction submodule, used to replace the abnormal score value with a preset score value, and use the replaced abnormal data source score as a corrected data source score;

[0108] The processed data source score combination submodule is used to obtain the processed data source score according to the modified data source score and the normal data source score.

[0109] The working principle of the above technical solution is as follows: after the data preprocessing module receives the data source score sent by the data source submodule, the score value acquisition submodule obtains the score value corresponding to the data source score, and then the score value discrimination submodule performs numerical analysis on the score value of each data source, distinguishes it into normal score value and abnormal score value, and uses the data source score corresponding to the normal score value as the normal data source score, and uses the data source score corresponding to the abnormal score value as the abnormal data source score; then, the abnormal type determination submodule determines the abnormal type of the score value, and the abnormal data source score correction submodule replaces all abnormal score values ​​with preset values ​​to obtain the corrected data source score. After that, the processing data source score combination submodule preprocesses the data, and uses the normal data source score and the corrected data source score as the processed data source score.

[0110] The beneficial effects of the above technical solution are: numerical analysis is performed on the data source scores, and the values ​​are divided into abnormal score values ​​and normal score values, which is convenient for data correction in the next step. According to the characteristics of the abnormal score value, its abnormal type is determined to facilitate the investigation and resolution of the problem of the third-party data information platform or data source. After determining the abnormal score value, it is filled with the preset value to ensure the capacity of the data source score sample, which is convenient for the next call.

[0111] According to some embodiments of the present invention, the method in which the monitoring technical indicator calculation module calculates the rate of change of the real-time technical indicator of the processing data source score compared to the historical technical indicator of the processing data source score includes:

[0112] The number of the normal data source scores is used as the effective call volume of the data source corresponding to the normal data source score by the third-party data information platform;

[0113] The number of processed data source scores is used as the total number of data source calls corresponding to the processed data source scores by the third-party data information platform;

[0114] Calculate the ratio of the effective call volume to the total call volume as the effective call rate;

[0115] The real-time technical indicator is formed according to the plurality of normal scoring values, the plurality of data source scoring values ​​and the effective call rate;

[0116] Obtaining historical technical indicators for processing data source scoring; the historical technical indicators include a number of historical normal scoring values, a number of historical data source scoring values, and a historical effective call rate;

[0117] Determine a first difference value according to the plurality of normal scoring values ​​and the plurality of historical normal scoring values, and determine a first change rate according to the first difference value and the plurality of historical normal scoring values;

[0118] Determine a second difference value according to the score values ​​of the plurality of data sources and the score values ​​of the plurality of historical data sources, and determine a second change rate according to the second difference value and the score values ​​of the plurality of historical data sources;

[0119] Calculating a third difference between the effective call rate and the historical effective call rate, and taking a ratio of the third difference to the historical effective call rate as a third change rate;

[0120] According to the first change rate, the second change rate and the third change rate, a change rate of the real-time technical indicator of the processing data source score compared to the historical technical indicator of the processing data source score is obtained.

[0121] The beneficial effects of the above technical solution are as follows: by analyzing the total normal score of the data source score, the overall status of the data source corresponding to the normal score data source in the process of interacting with the third-party data information platform can be determined. Similarly, by analyzing the total value of the data source score, the overall status of all data sources in the process of interacting with the third-party data information platform can be analyzed. By analyzing the effective call rate, the proportion of normal data source scores in all data source scores can be determined. By analyzing the change rate of real-time technical indicators and historical technical indicators, the trend of data source scores changing over time can be determined.

[0122] According to some embodiments of the present invention, the warning rule module includes:

[0123] A first historical data acquisition submodule, used to acquire first historical data of a change rate of a real-time technical indicator of a score of a data source to be analyzed compared to a historical technical indicator of a score of a data source to be analyzed;

[0124] A change rate time series establishment submodule is used to set a sampling period to sample the first historical data of the change rate, obtain a plurality of change rate sampling values, and establish a change rate time series of the change rate sampling values ​​with respect to time;

[0125] A confidence interval determination submodule, used to determine the confidence interval of the change rate sampling value in the change rate time series;

[0126] A sampling point discrimination submodule is used to mark a number of sampling points in the change rate time series that are not within the confidence interval as suspicious sampling points, take the time point where each suspicious sampling point is located as a suspicious time point, take the time points of a number of non-suspicious time points adjacent to each suspicious time point as verification time points, and take the sampling point corresponding to each verification time point as a verification sampling point;

[0127] A suspicious sampling value acquisition submodule is used to obtain the suspicious sampling value corresponding to each suspicious sampling point;

[0128] A verification sampling value acquisition submodule is used to obtain the verification sampling value of the verification sampling point corresponding to each suspicious sampling point;

[0129] An outlier index calculation and comparison submodule, used to calculate the outlier index of the suspicious sampling point according to the suspicious sampling value and a number of verification sampling values ​​corresponding to the suspicious sampling value, and compare it with the outlier index threshold;

[0130] The abnormal sampling point marking submodule is used to mark the suspicious sampling point as an abnormal sampling point when the outlier index calculation and comparison submodule determines that the outlier index is greater than the outlier index threshold, and mark the suspicious time point corresponding to the abnormal sampling point as an abnormal time point, and obtain a change rate time series containing abnormal marks as the abnormal change rate time series;

[0131] The prediction model building submodule is used to build a prediction model for abnormal change rate time series based on neural network technology;

[0132] A prediction model training submodule, used to train the prediction model using the abnormal change rate time series as training data to obtain a target prediction model;

[0133] The submodule for determining the data source to be analyzed is used to select any data source score from a number of data source scores as the data source score to be analyzed;

[0134] A change rate prediction monitoring submodule is used to obtain historical data of the change rate of the real-time technical indicators of the data source score to be analyzed compared with the historical technical indicators of the data source score to be analyzed, and input it into the target prediction model for monitoring and early warning;

[0135] The outlier index calculation and comparison submodule calculates the outlier index of the suspicious sampling point according to the suspicious sampling value and a number of verification sampling values ​​corresponding to the suspicious sampling value, including:

[0136] Establish a rectangular coordinate system to obtain the coordinate information of the suspicious sampling points and the verification sampling points;

[0137] Substituting the coordinate information of the suspicious sampling point and a number of verification sampling points corresponding to the suspicious sampling point into an outlier index formula to obtain an outlier index of the suspicious sampling point;

[0138] The outlier index formula is:

[0139]

[0140] Wherein, E is the outlier index of the suspicious sampling point; n is the number of verification sampling points corresponding to the suspicious sampling point; x is the horizontal coordinate of the suspicious sampling point; y is the vertical coordinate of the suspicious sampling point; x kis the horizontal coordinate of the kth verification sampling point corresponding to the suspicious sampling point; k is the ordinate of the kth verification sampling point corresponding to the suspicious sampling point; max(x k ) is the maximum value among the abscissas of the plurality of verification sampling points; min(x k ) is the minimum value among the abscissas of the plurality of verification sampling points; max(y k ) is the maximum value among the vertical coordinates of the plurality of verification sampling points; min(y k ) is the minimum value among the vertical coordinates of the several verification sampling points.

[0141] The working principle of the above technical solution is as follows: the first historical data acquisition submodule is used to obtain the first historical data of the change rate of the real-time technical indicators of the data source score to be analyzed compared with the historical technical indicators of the data source score to be analyzed; then, the change rate time series establishment submodule collects and samples the first historical data of the technical indicator change rate, obtains several sampling values ​​of the change rate, and establishes the change rate time series in combination with the time points corresponding to the change rate sampling values; the confidence interval determination submodule determines the confidence interval of the change rate sampling value according to the statistical principle, and then marks the sampling points that are not in the interval as suspicious sampling points through the sampling point discrimination submodule, and the time points corresponding to the suspicious sampling points are used as suspicious time points. Several non-suspicious time points near the suspicious time points are used as verification time points, and the suspicious sampling value acquisition submodule and the verification sampling value acquisition submodule respectively obtain the suspicious sampling point value and the corresponding verification sampling point value; then, the outlier index calculation and comparison submodule places the suspicious sampling point and its corresponding verification sampling point into a rectangular coordinate system, and calculates the outlier index of the suspicious sampling point according to its coordinate value. After obtaining the outlier index, it is compared with the preset outlier index threshold. When the outlier index calculation and comparison submodule determines that the outlier index is greater than the threshold, the abnormal sampling point marking submodule identifies the suspicious sampling point as an abnormal sampling point, marks it, and obtains the abnormal change rate time series containing the abnormal mark. After that, the prediction model establishment submodule uses neural network technology to build a prediction model, and then the prediction model training submodule uses the abnormal change rate time series as training data to train the prediction model to obtain a target prediction model that can predict the change rate time series. After that, the data source determination submodule to be analyzed selects at least one data source score from several data source scores as the data source score to be analyzed, obtains the data corresponding to the change rate of the technical indicators of the data source score to be analyzed, and inputs it into the target prediction model. The change rate prediction and monitoring submodule predicts the change trend of the change rate of its real-time technical indicators compared with the change rate of the historical technical indicators, and monitors and warns according to the predicted change trend.

[0142] The beneficial effects of the above technical solutions are as follows: Analyze the historical data of the change rate from the perspective of time to obtain the corresponding relationship between the change rate and time, which is conducive to re-analyzing the change of the change rate, further helping business personnel to control the trend of the change rate and determine the abnormal time point of the change rate; divide the sampling points according to whether they are within the confidence interval, so that most of the sampling points fall within the confidence interval, and only a few sampling points are outside the confidence interval, ensuring that the number of suspicious sampling points that need to calculate the outlier index is small, reducing the amount of calculation. The outlier index, an indicator that combines the time point factor and the sampling value factor, is used to determine whether a suspicious sampling point is an abnormal sampling point, which overcomes the inherent defects of the traditional abnormal analysis of suspicious sampling points based only on a single indicator such as the time point or the sampling value, making the reference significance of the judgment result higher; analyze the change rate of the technical indicators of the data source scoring to be analyzed according to the prediction model trained based on the neural network technology, which has a high degree of automation and can save human resources.

[0143] like Figure 3 As shown, the present invention provides a change rate prediction monitoring submodule, including:

[0144] A second historical data determination unit, configured to obtain second historical data of a change rate of the real-time technical indicator of the score of the data source to be analyzed compared to the historical technical indicator of the score of the data source to be analyzed;

[0145] A first sampling unit is used to sample the second historical data to obtain a second historical data sequence;

[0146] A second sampling unit, used for sampling the preceding data of the second historical data to obtain a preceding data sequence;

[0147] A first prediction data sequence determining unit, configured to input the preceding data sequence into the target prediction model for data analysis and prediction, and obtain a first prediction data sequence including a plurality of prediction data;

[0148] A first correlation coefficient calculation and comparison unit, used to calculate a first correlation coefficient between the second historical data sequence and the first predicted data sequence according to a correlation coefficient calculation formula, and compare the first correlation coefficient with a preset first correlation coefficient threshold;

[0149] a second prediction data sequence determining unit, configured to input the second historical data sequence into the target prediction model to obtain a second prediction data sequence when the first correlation coefficient calculating and comparing unit determines that the first correlation coefficient is greater than the first correlation coefficient threshold;

[0150] A third prediction data sequence determining unit, configured to input the first prediction data sequence into the target prediction model to obtain a third prediction data sequence;

[0151] A second correlation coefficient calculation and comparison unit, used to calculate a second correlation coefficient between the second prediction data series and the third prediction data series according to a correlation coefficient calculation formula, and compare the second correlation coefficient with a preset second correlation coefficient threshold;

[0152] a prediction abnormality index calculation and comparison unit, configured to perform abnormality analysis on the second prediction data sequence to obtain a prediction suspicious data point and a corresponding prediction outlier index in the second prediction data sequence when the second correlation coefficient calculation and comparison unit determines that the second correlation coefficient is greater than the second correlation coefficient threshold, and compare the prediction outlier index with a preset prediction outlier index threshold;

[0153] The change rate abnormality judgment and early warning unit is used to judge that the change trend of the change rate of the real-time technical indicators of the data source score to be analyzed compared with the historical technical indicators of the data source score to be analyzed is abnormal, and issue an early warning when the predicted outlier index calculation and comparison unit determines that the predicted outlier index is greater than a preset predicted outlier index threshold.

[0154] The working principle of the above technical scheme: when the change rate prediction and monitoring submodule monitors the change rate in the above scheme, the second historical data determination unit obtains the second historical data of the technical indicators of the data source score to be analyzed, and the second historical data is a plurality of continuous data of the technical indicators of the data source score to be analyzed within a certain time period; the first sampling unit and the second sampling unit sample the second historical data and the preceding data respectively to obtain a second historical data sequence and a preceding data sequence; thereafter, the first predicted data sequence determination unit inputs the preceding data sequence into the target prediction model to obtain a first predicted data sequence of the preceding data sequence; thereafter, the first correlation coefficient calculation and comparison unit calculates the first correlation coefficient between the first predicted data sequence and the preceding data sequence according to the Pearson correlation coefficient calculation formula, and compares it with a preset first correlation coefficient threshold. When the first correlation coefficient calculation and comparison unit determines that the first correlation coefficient is greater than the first correlation coefficient threshold, the second prediction data sequence determination unit and the second prediction data sequence determination unit input the second historical data sequence into the target prediction model to predict the trend of the second historical data sequence to obtain a second prediction data sequence; at the same time, the third prediction data sequence determination unit inputs the first prediction data sequence into the target prediction model for prediction to obtain a third prediction data sequence; the second correlation coefficient calculation and comparison unit calculates the second correlation coefficient between the second prediction data sequence and the third prediction data sequence, and compares it with the preset second correlation coefficient threshold. When the prediction anomaly index calculation and comparison unit determines that the second correlation coefficient is greater than the second correlation coefficient threshold in the second correlation coefficient calculation and comparison unit, it means that the correlation between the second prediction data sequence and the third prediction data sequence is high, and the target prediction model is more accurate in predicting the trend of the historical data sequence; thereafter, the change rate anomaly judgment and early warning unit calculates the prediction outlier index of the prediction suspicious data point in the second prediction data sequence according to the anomaly analysis described above, and compares it with the preset prediction outlier index threshold. When it is determined that the prediction outlier index is greater than the prediction outlier index threshold, it means that there is a possibility of anomalies in the second historical data sequence in the future, that is, there is an abnormality in the change trend of the change rate of the technical indicator of the data source score to be analyzed, and an early warning is issued for the second historical data sequence.

[0155] The beneficial effects of the above technical solution are as follows: before performing predictive analysis on the second historical data sequence, predicting the preceding data of the second historical data sequence to obtain a first predicted data sequence, calculating the correlation coefficient between the first predicted data sequence and the second historical data sequence, and verifying the data prediction effect of the target prediction model based on the correlation coefficient, thereby ensuring the reliability of the prediction of the second historical data sequence by the target prediction model; after predicting the second historical data sequence to obtain the second predicted data sequence, predicting the first predicted data sequence to obtain a third predicted data sequence, and calculating the second correlation coefficient between the second predicted data sequence and the third predicted data sequence. This step is a secondary verification of the data prediction effect of the target prediction model, thereby further ensuring the reliability of the prediction of the second historical data sequence by the target prediction model, and ensuring that the second predicted data sequence is the future trend of the second historical data sequence.

[0156] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A non-real-time data monitoring and early warning method for a third-party data information platform, characterized in that: include: Collect data source scores of several data sources from third-party data information platforms; Preprocessing the data source score to obtain a processed data source score; Calculate the change rate of the real-time technical indicator of the processing data source score compared to the historical technical indicator of the processing data source score, and monitor and warn the change rate; When determining that the change rate is abnormal, determining the data product corresponding to the change rate abnormality, and generating a list of abnormal products; Visualizing the list of abnormal products; The preprocessing of the data source score to obtain the processed data source score includes: Obtaining a plurality of data source score values ​​corresponding to the plurality of data source scores, each data source score corresponding to a data source score value; Performing numerical analysis on the score values ​​of the plurality of data sources to determine a plurality of abnormal score values ​​and a plurality of normal score values ​​among the score values ​​of the plurality of data sources; The data source score corresponding to the normal score value is used as the normal data source score; The data source score corresponding to the abnormal score value is used as the abnormal data source score; Determining the abnormal type of the abnormal data source score according to the abnormal score value; The abnormal score value is replaced with a preset score value, and the replaced abnormal data source score is used as the modified data source score; Obtaining the processed data source score according to the modified data source score and the normal data source score; Calculating the change rate of the real-time technical indicator of the processing data source score compared to the historical technical indicator of the processing data source score, including: The number of the normal data source scores is used as the effective call volume of the data source corresponding to the normal data source score by the third-party data information platform; The number of processed data source scores is used as the total number of data source calls corresponding to the processed data source scores by the third-party data information platform; Calculate the ratio of the effective call volume to the total call volume as the effective call rate; The real-time technical indicator is formed according to the plurality of normal scoring values, the plurality of data source scoring values ​​and the effective call rate; Obtaining historical technical indicators for processing data source scoring; the historical technical indicators include a number of historical normal scoring values, a number of historical data source scoring values, and a historical effective call rate; Determine a first difference value according to the plurality of normal scoring values ​​and the plurality of historical normal scoring values, and determine a first change rate according to the first difference value and the plurality of historical normal scoring values; Determine a second difference value according to the score values ​​of the plurality of data sources and the score values ​​of the plurality of historical data sources, and determine a second change rate according to the second difference value and the score values ​​of the plurality of historical data sources; Calculating a third difference between the effective call rate and the historical effective call rate, and taking a ratio of the third difference to the historical effective call rate as a third change rate; According to the first change rate, the second change rate and the third change rate, a change rate of the real-time technical indicator of the processing data source score compared to the historical technical indicator of the processing data source score is obtained.

2. The non-real-time data monitoring and early warning method of the third-party data information platform according to claim 1 is characterized in that: Monitoring and early warning of the change rate include: Acquire first historical data of a change rate of a real-time technical indicator of a score of a data source to be analyzed compared to a historical technical indicator of a score of the data source to be analyzed; Setting a sampling period to sample the first historical data of the change rate, obtaining a plurality of change rate sampling values, and establishing a change rate time series of the change rate sampling values ​​with respect to time; Determining a confidence interval for the rate of change sample values ​​in the rate of change time series; Marking a number of sampling points in the change rate time series that are not within the confidence interval as suspicious sampling points, taking the time point at which each suspicious sampling point is located as a suspicious time point, taking a number of non-suspicious time points adjacent to each suspicious time point as verification time points, and taking the sampling point corresponding to each verification time point as a verification sampling point; Obtain the suspicious sampling value corresponding to each suspicious sampling point; Obtain the verification sampling value of the verification sampling point corresponding to each suspicious sampling point; Calculate the outlier index of the suspicious sampling point according to the suspicious sampling value and a number of verification sampling values ​​corresponding to the suspicious sampling value, and compare it with the outlier index threshold; When it is determined that the outlier index is greater than the outlier index threshold, the suspicious sampling point is marked as an abnormal sampling point, and the suspicious time point corresponding to the abnormal sampling point is marked as an abnormal time point, and a change rate time series containing abnormal marks is obtained as an abnormal change rate time series; Establish a prediction model for abnormal change rate time series; Using the abnormal change rate time series as training data to train the prediction model, to obtain a target prediction model; Among several data source scores, one data source score is randomly selected as the data source score to be analyzed; The historical data of the change rate of the real-time technical indicator of the score of the data source to be analyzed compared with the historical technical indicator of the score of the data source to be analyzed is obtained, and input into the target prediction model for monitoring and early warning.

3. The non-real-time data monitoring and early warning method of the third-party data information platform as claimed in claim 2 is characterized in that: Obtaining historical data of the change rate of the real-time technical indicator of the score of the data source to be analyzed compared with the historical technical indicator of the score of the data source to be analyzed, and inputting it into the target prediction model for monitoring and early warning, including: Acquire second historical data of a change rate of the real-time technical indicator of the score of the data source to be analyzed compared with the historical technical indicator of the score of the data source to be analyzed; Sampling the second historical data to obtain a second historical data sequence; Sampling the preceding data of the second historical data to obtain a preceding data sequence; Inputting the preceding data sequence into the target prediction model for data analysis and prediction to obtain a first prediction data sequence including a plurality of prediction data; Calculate the first correlation coefficient between the second historical data sequence and the first predicted data sequence according to the correlation coefficient calculation formula, and compare the first correlation coefficient with a preset first correlation coefficient threshold; When it is determined that the first correlation coefficient is greater than the first correlation coefficient threshold, the second historical data sequence is input into the target prediction model to obtain a second prediction data sequence; the first prediction data sequence is input into the target prediction model to obtain a third prediction data sequence; Calculating a second correlation coefficient between the second prediction data series and the third prediction data series according to a correlation coefficient calculation formula, and comparing the second correlation coefficient with a preset second correlation coefficient threshold; When it is determined that the second correlation coefficient is greater than the second correlation coefficient threshold, performing an abnormality analysis on the second predicted data sequence to obtain predicted suspicious data points and corresponding predicted outlier indexes in the second predicted data sequence, and comparing the predicted outlier index with a preset predicted outlier index threshold; When it is determined that the predicted outlier index is greater than a preset predicted outlier index threshold, it is determined that the change trend of the real-time technical indicator of the data source score to be analyzed compared with the change trend of the historical technical indicator of the data source score to be analyzed is abnormal, and an early warning is issued.

4. A non-real-time data monitoring and early warning system for a third-party data information platform, characterized in that: include: The data source submodule is used to collect data source scores of several data sources from third-party data information platforms; A data preprocessing submodule, used to preprocess the data source score to obtain a processed data source score; A monitoring technical indicator calculation submodule is used to calculate the change rate of the real-time technical indicator of the processing data source score compared with the historical technical indicator of the processing data source score, and monitor and warn the change rate; The early warning rule submodule is used to determine the data product corresponding to the abnormal change rate when the change rate is determined to be abnormal, and generate a list of abnormal products; The early warning report display submodule is used to visualize the abnormal product list; The data preprocessing submodule includes: A scoring value acquisition submodule, used to acquire a plurality of data source scoring values ​​corresponding to the plurality of data source scores; The rating value discrimination submodule is used to: Performing numerical analysis on the score values ​​of the plurality of data sources to determine a plurality of abnormal score values ​​and a plurality of normal score values ​​among the score values ​​of the plurality of data sources; The data source score corresponding to the normal score value is used as the normal data source score; The data source score corresponding to the abnormal score value is used as the abnormal data source score; An anomaly type determination submodule, used to determine the anomaly type of the anomaly data source score according to the anomaly score value; An abnormal data source score correction submodule, used to replace the abnormal score value with a preset score value, and use the replaced abnormal data source score as a corrected data source score; A processed data source score combination submodule, used to obtain the processed data source score according to the modified data source score and the normal data source score; The method in which the monitoring technical indicator calculation module calculates the change rate of the real-time technical indicator of the processing data source score compared with the historical technical indicator of the processing data source score includes: The number of the normal data source scores is used as the effective call volume of the data source corresponding to the normal data source score by the third-party data information platform; The number of processed data source scores is used as the total number of data source calls corresponding to the processed data source scores by the third-party data information platform; Calculate the ratio of the effective call volume to the total call volume as the effective call rate; The real-time technical indicator is formed according to the plurality of normal scoring values, the plurality of data source scoring values ​​and the effective call rate; Obtaining historical technical indicators for processing data source scoring; the historical technical indicators include a number of historical normal scoring values, a number of historical data source scoring values, and a historical effective call rate; Determine a first difference value according to the plurality of normal scoring values ​​and the plurality of historical normal scoring values, and determine a first change rate according to the first difference value and the plurality of historical normal scoring values; Determine a second difference value according to the score values ​​of the plurality of data sources and the score values ​​of the plurality of historical data sources, and determine a second change rate according to the second difference value and the score values ​​of the plurality of historical data sources; Calculating a third difference between the effective call rate and the historical effective call rate, and taking a ratio of the third difference to the historical effective call rate as a third change rate; According to the first change rate, the second change rate and the third change rate, a change rate of the real-time technical indicator of the processing data source score compared to the historical technical indicator of the processing data source score is obtained.

5. The non-real-time data monitoring and early warning system of the third-party data information platform as claimed in claim 4 is characterized in that: The warning rule submodule includes: A first historical data acquisition submodule, used to acquire first historical data of a change rate of a real-time technical indicator of a score of a data source to be analyzed compared to a historical technical indicator of a score of a data source to be analyzed; A change rate time series establishment submodule is used to set a sampling period to sample the first historical data of the change rate, obtain a plurality of change rate sampling values, and establish a change rate time series of the change rate sampling values ​​with respect to time; A confidence interval determination submodule, used to determine the confidence interval of the change rate sampling value in the change rate time series; A sampling point discrimination submodule is used to mark a number of sampling points in the change rate time series that are not within the confidence interval as suspicious sampling points, take the time point where each suspicious sampling point is located as a suspicious time point, take the time points of a number of non-suspicious time points adjacent to each suspicious time point as verification time points, and take the sampling point corresponding to each verification time point as a verification sampling point; A suspicious sampling value acquisition submodule is used to obtain the suspicious sampling value corresponding to each suspicious sampling point; A verification sampling value acquisition submodule is used to obtain the verification sampling value of the verification sampling point corresponding to each suspicious sampling point; An outlier index calculation and comparison submodule, used to calculate the outlier index of the suspicious sampling point according to the suspicious sampling value and a number of verification sampling values ​​corresponding to the suspicious sampling value, and compare it with the outlier index threshold; The abnormal sampling point marking submodule is used to mark the suspicious sampling point as an abnormal sampling point when the outlier index calculation and comparison submodule determines that the outlier index is greater than the outlier index threshold, and mark the suspicious time point corresponding to the abnormal sampling point as an abnormal time point, and obtain a change rate time series containing abnormal marks as the abnormal change rate time series; The prediction model building submodule is used to build a prediction model for the abnormal change rate time series; A prediction model training submodule, used to train the prediction model using the abnormal change rate time series as training data to obtain a target prediction model; The submodule for determining the data source to be analyzed is used to select any data source score from a number of data source scores as the data source score to be analyzed; The change rate prediction monitoring submodule is used to obtain historical data of the change rate of the real-time technical indicators of the data source score to be analyzed compared with the historical technical indicators of the data source score to be analyzed, and input it into the target prediction model for monitoring and early warning.

6. The non-real-time data monitoring and early warning system for a third-party data information platform as claimed in claim 5, characterized in that: The change rate prediction and monitoring submodule includes: A second historical data determination unit, configured to obtain second historical data of a change rate of the real-time technical indicator of the score of the data source to be analyzed compared to the historical technical indicator of the score of the data source to be analyzed; A first sampling unit is used to sample the second historical data to obtain a second historical data sequence; A second sampling unit, used for sampling the preceding data of the second historical data to obtain a preceding data sequence; A first prediction data sequence determining unit, configured to input the preceding data sequence into the target prediction model for data analysis and prediction, and obtain a first prediction data sequence including a plurality of prediction data; A first correlation coefficient calculation and comparison unit, used to calculate a first correlation coefficient between the second historical data sequence and the first predicted data sequence according to a correlation coefficient calculation formula, and compare the first correlation coefficient with a preset first correlation coefficient threshold; a second prediction data sequence determining unit, configured to input the second historical data sequence into the target prediction model to obtain a second prediction data sequence when the first correlation coefficient calculating and comparing unit determines that the first correlation coefficient is greater than the first correlation coefficient threshold; A third prediction data sequence determining unit, configured to input the first prediction data sequence into the target prediction model to obtain a third prediction data sequence; A second correlation coefficient calculation and comparison unit, used to calculate a second correlation coefficient between the second prediction data series and the third prediction data series according to a correlation coefficient calculation formula, and compare the second correlation coefficient with a preset second correlation coefficient threshold; a prediction abnormality index calculation and comparison unit, configured to perform abnormality analysis on the second prediction data sequence to obtain a prediction suspicious data point and a corresponding prediction outlier index in the second prediction data sequence when the second correlation coefficient calculation and comparison unit determines that the second correlation coefficient is greater than the second correlation coefficient threshold, and compare the prediction outlier index with a preset prediction outlier index threshold; The change rate abnormality judgment and early warning unit is used to judge that the change trend of the change rate of the real-time technical indicators of the data source score to be analyzed compared with the historical technical indicators of the data source score to be analyzed is abnormal, and issue an early warning when the predicted outlier index calculation and comparison unit determines that the predicted outlier index is greater than a preset predicted outlier index threshold.

Images