Data supervision system and method based on unified data exchange service

Abstract

The present invention discloses a data supervision system and method based on a unified data exchange service, and belongs to the field of technical data supervision. The method comprises the following steps: S100, collecting data in which errors occurred during transmission by various departments in history, and extracting characteristic factors that cause errors after analyzing the data; S200, calculating a threshold for judging whether an error occurred in the transmission data according to the characteristic factor values ​​of each department in history when transmitting data; S300, finding a credible parameter that has an influencing relationship with the characteristic factor, and calculating a credibility threshold for judging data error warning; S400, using the error threshold to judge whether an error occurred in the real-time data transmitted by each department; S500, calculating the credibility of the error warning issued in real time, and judging whether the error warning is credible according to the credibility threshold. The present invention obtains characteristic factors that affect data errors by analyzing and calculating, analyzes the characteristic factors when receiving the data, prevents data errors, and performs data backtracking in time.

Description

Technical Field

[0001] The present invention relates to the technical field of data supervision, and in particular to a data supervision system and method based on a unified data exchange service. Background Art

[0002] The unified data exchange service is an integrated technology platform designed to achieve efficient, secure and standardized data exchange between departments. Through this platform, the financial department can better manage, analyze and utilize cross-departmental data to optimize policy formulation, improve resource allocation efficiency, and strengthen cross-departmental collaboration.

[0003] In the existing technical background, with the development of data exchange services and the widespread promotion of applications, data exchange between multiple departments and systems is becoming increasingly frequent. However, due to the diversity of data sources, the heterogeneity of data formats, and the instability of the network environment, various errors often occur during data transmission. These errors may include data loss, data corruption, data delay, data transmission interruption, etc., which seriously affect the normal operation of the system and the accuracy of the data. In the face of a large number of system departments and massive amounts of data, data transmission errors, without an automated error warning mechanism, make it difficult for decision-makers to detect data errors in a short time, and failure to trace data in a short time will lead to decision-making errors. Summary of the invention

[0004] The purpose of the present invention is to provide a data supervision system and method based on a unified data exchange service to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the present invention provides the following technical solution: a data supervision method based on a unified data exchange service, the method comprising the following steps:

[0006] S100, collecting data on errors that occurred during transmission by various departments in history, analyzing the data and extracting characteristic factors that caused the errors;

[0007] S200, calculating a threshold value for determining whether an error occurs in data transmission based on the characteristic factor values ​​of each department in the history of data transmission;

[0008] S300, extracting historical data records of errors in data transmission by various departments, analyzing the relationship between all data and characteristic factors, finding credible parameters that have an influencing relationship with the characteristic factors, and calculating the credibility threshold of data error warning judgment;

[0009] S400, for the real-time transmitted data, using the error threshold to determine whether the real-time data transmitted by each department has an error, and issuing an error warning in real time;

[0010] S500: Calculate the credibility of the error warning issued in real time, and determine whether the error warning is credible according to the credibility threshold.

[0011] In step S100, the specific steps of extracting the characteristic factors causing errors after analyzing the data are as follows:

[0012] S101, collect data {Q1, Q2, Q3, ..., Qn} of errors in transmission of each department in history; Q1, Q2, Q3, ..., Qn represent the 1st, 2nd, 3rd, ..., nth error data values ​​in transmission of each department in history, and n is a positive integer; collect data of errors in transmission of each department in history m times; and collect data of no errors in transmission n times, and collect m times in the same way;

[0013] S102, collect n data of errors and no errors in the transmission of each department in the m times of history, and calculate the average value of the change when each department transmits data, the formula is:

[0014]

[0015] Among them, Q(p) represents the average value of the change when each department transmits data, Q' represents the data value without error when each department transmits data, Q represents the data value with error when each department transmits data, n represents the number of collected data, and M represents the number of influencing factors of data error; the influencing factors include data transmission delay, data update frequency and data recording time deviation, etc.

[0016] S103. Calculate the average value of the change in transmission data of each department as {Q(p1), Q(p2), Q(p3), ..., Q(pl)}; Q(p1), Q(p2), Q(p3), ..., Q(pl) represent the calculated average values ​​of the change in transmission data of the 1st, 2nd, 3rd ... lth departments, conduct comparative analysis on the average values ​​of the change in l departments, extract the departments whose average values ​​of transmission data change are greater than 50%, count the influencing factors that cause data errors, and record the influencing factors with the largest number as characteristic factors; check the characteristic factor values ​​T(Qk) of each department when transmitting data, where k represents the characteristic labels of different characteristic factors.

[0017] In step S200, the specific steps of calculating the threshold value for determining the error in data transmission of each department are:

[0018] S201, collect the characteristic factor values ​​of each department in the history of data transmission {{T(Q1 k1 ), T(Q1 k2 ), T(Q1 k3 ), ..., T(Q1 kj )},{T(Q2k1 ), T(Q2 k2 ), T(Q2 k3 ), ..., T(Q2 kj )}, ..., {T(Qm k1 ), T(Qm k2 ), T(Qm k3 ), ..., T(Qm kj )}}; T(Q1), T(Q2), T(Q3), ..., T(Qm) represent the characteristic factor values ​​of the collected transmission data when an error occurs for the 1st, 2nd, 3rd ... mth time, m is a positive integer, { k1 , k2 , k3 , ..., kj} respectively represent the feature labels of different feature factors;

[0019] S202. Calculate the error threshold according to the characteristic factor values ​​of errors that occurred in data transmission in each department in history. The formula is:

[0020]

[0021] Among them, r represents the error threshold for judging the error in data transmission of each department, T(Qu kv ) represents the characteristic factor value of the collected transmission data when an error occurs.

[0022] In step S300, the specific steps of searching for a credible parameter that has an influencing relationship with a characteristic factor are:

[0023] S311. Collect K influencing factors (current network speed and department data update frequency, etc.) when characteristic factors change during data transmission in each department in history, draw a linear graph with the influencing factors as independent variables and the characteristic factors as dependent variables, observe and analyze the drawn linear graph, establish a regression model for each influencing factor and characteristic factor, and use the regression model to represent the relationship between each influencing factor and characteristic factor; the calculation expression is:

[0024] T″(Q k )=a×h+b

[0025] Among them, T”(Q k ) represents the predicted value of the characteristic factor, h represents the value of the influencing factor, a represents the slope of the regression model, and b represents the intercept of the regression model;

[0026] S312. Calculate the slope and intercept of the regression model. The specific formula is:

[0027]

[0028] b=T′(Qf )-a×h′(f)

[0029] Among them, e represents the number of collected influencing factors and each characteristic factor, h(f) represents the value of each influencing factor, T(Q f ) represents the different characteristic factor values ​​collected, h′(f) represents the average value of the influencing factor, T′(Qf) represents the average value of the characteristic factor, f∈[1,e]; the expressions of the regression model of the K characteristic factors are calculated in turn, and the predicted values ​​of the K characteristic factors are finally obtained as {T”(Q k )_1,T”(Q k )_2,T”(Q k )_3, ..., T”(Q k )_K};T”(Q k )_1,T”(Q k )_2,T”(Q k )_3, ..., T”(Q k )_K represent the predicted values ​​of the 1st, 2nd, 3rd, ..., Kth characteristic factors respectively;

[0030] S313. After calculating the predicted value of each characteristic factor, the difference between the predicted value and the true value of each characteristic factor is calculated. The formula is:

[0031]

[0032] Among them, ch represents the difference between the predicted value and the true value of the characteristic factor in each regression model, T(Q k ) represents the true value of the characteristic factor of each point, T”(Q k ) represents the predicted value of the characteristic factor, and e represents the number of characteristic factor values; after calculation, the K difference values ​​are {ch_1, ch_2, ch_3, ..., ch_K}, where ch_1, ch_2, ch_3, ..., ch_K represent the calculated difference values ​​between the 1st, 2nd, 3rd, ..., Kth characteristic factor predicted values ​​and the true values, respectively;

[0033] S314: Calculate the median value of the K difference values, specifically in the following manner:

[0034] Sort ch_1, ch_2, ch_3, ..., ch_K in descending order, and take Differences is the median value; when When , it is judged that the characteristic factor and the influencing factor corresponding to the difference value ch do not have an influence relationship. When , it is judged that the characteristic factor corresponding to the difference value ch and the influencing factor have an influence relationship, and the influencing factor with an influence relationship is defined as the credible factor B.

[0035] For each environmental data, a regression model is constructed. By calculating the difference values of each regression model, it can be determined whether there is an influence relationship between the corresponding influencing factors and characteristic factors. The credible parameters among the influencing factors that have an influence relationship with the characteristic factors are found. Using the credible parameters, it can be verified whether the early warning judgment is credible, greatly improving the credibility of the system's early warning judgment and reducing the probability of judgment errors;

[0036] In step S300, the specific steps for calculating the credibility threshold of the data error early warning judgment are as follows:

[0037] S321. The credible parameters that have an influence on the characteristic factors through calculation are {B1, B2, B3,..., Bd}, where B1, B2, B3,..., Bd represent the 1st, 2nd, 3rd,..., dth credible parameters obtained through calculation, and d is a positive integer; collect each credible parameter value when data errors occur in the data of each department in history;

[0038] S322. Calculate the credibility threshold of the error early warning according to the credible parameter values. The formula is:

[0039]

[0040] where re represents the credibility threshold of the data early warning when a data error occurs, B represents the credible parameter value when each department transmits data, and d represents the number of credible parameters.

[0041] In step S400, the specific method for using the error threshold to determine whether the real-time data transmitted by each department has an error is as follows:

[0042] For the characteristic factor value T(D) of the real-time transmitted data, use the error threshold calculated in S200 to judge the real-time characteristic factor value. When T(D) ≥ r, it is judged that the real-time transmitted data has an error and an error early warning is issued. When T(D) < r, it is judged that the real-time transmitted data is normal.

[0043] In step S500, the specific steps for calculating the credibility of the error early warning issued in real time are as follows:

[0044] S501. After judging the real-time transmitted data and issuing an early warning, extract the regression models of each credible parameter and the characteristic factor in S312, and calculate all real-time credible parameter values using the real-time characteristic factor value T(D). Specifically:

[0045] Bs = a × T(D) + b

[0046] Calculate each real-time trust parameter value as {Bs1, Bs2, Bs3,..., Bsd}, where Bs1, Bs2, Bs3,..., Bsd represent the 1st, 2nd, 3rd,..., dth real-time trust parameter values respectively;

[0047] S502. Calculate the credibility of the error warning based on the real-time trust parameter values. The formula is:

[0048]

[0049] In the formula, re′ represents the credibility of the error warning calculated in real time, Bs represents each real-time calculated trust parameter value, and d represents the number of trust parameters;

[0050] S503. Extract the credibility threshold calculated in S322 to judge the credibility of the real-time error warning. When re′ < re, it is judged that the credibility of the error warning is not high and no warning operation is performed; when re′ ≥ re, it is judged that the error warning is credible, a data anomaly warning is issued, and the data source is immediately checked and the data is retransmitted.

[0051] A data supervision system based on a unified data exchange service, which includes a historical data collection and error analysis module, an error threshold calculation module, a feature relationship analysis module, a real-time transmission monitoring and error warning module, and a warning credibility calculation and management module;

[0052] The historical data collection and error analysis module is responsible for collecting the historical data of errors occurring during the transmission of each department, cleaning and analyzing these data, and extracting the characteristic factors leading to the transmission errors;

[0053] The error threshold calculation module is used to calculate the threshold for judging whether real-time data transmission has errors according to the characteristic factors extracted from the historical data;

[0054] The feature relationship analysis module is used to extract the data records of transmission errors in history, analyze the relationship between these data and the characteristic factors, determine the trust parameters affecting the transmission errors, and calculate the credibility threshold of the error warning;

[0055] The real-time transmission monitoring and error warning module is used to monitor the real-time data transmission of each department, evaluate in real time whether there is an error risk in the data according to the calculated error threshold, and issue an error warning in a timely manner;

[0056] The warning credibility calculation and management module is used to calculate the credibility of the real-time error warning, and judge the reliability of the warning based on the credibility threshold; if the credibility of the warning exceeds the threshold, the warning is considered credible and further processing is performed.

[0057] The historical data collection and error analysis module includes a historical data collection unit, a metadata management unit, a data exchange unit and an error analysis unit;

[0058] The data collection unit is used to collect data records transmitted by various departments in history; the metadata management unit is used to record the source department of the data, the sending and receiving time and the data type, and to track the source of the data and resend it after the system issues an error warning; the data exchange unit is used to use the external network-intranet cross method to exchange data, and the internal network is used for data transmission between each department and the supervision system. After the supervision system makes a decision on the received data, it uses the external network to send the decision to the destination departments; the error analysis unit is used to extract key factors in historical data and then perform error analysis on the data.

[0059] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: the present invention analyzes a large amount of data transmitted by various departments in history, finds characteristic factors that may cause data transmission errors, analyzes a large amount of transmission data records, and compares to find out which influencing factors have changed during the transmission of data with errors and data without errors, and further analyzes them as influencing factors to find the error threshold at which the influencing factor causes data errors. To ensure the reliability of the error threshold, the historical data is further analyzed, and the credibility threshold of the error threshold is calculated. The double guarantee ensures the reliability of the data error warning mechanism. When data errors occur, a quick real-time warning can be issued, data backtracking can be performed, and data can be resent to ensure the correctness of the data received when making decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0061] Figure 1 It is a schematic diagram of the steps of the data supervision method based on the unified data exchange service of the present invention;

[0062] Figure 2 It is a line graph of characteristic factors-influencing factors of the data supervision system and method based on unified data exchange service of the present invention. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] See also Figure 1-Figure 2 The present invention provides a technical solution: a data supervision method based on a unified data exchange service, the method comprising the following steps:

[0065] S100, collecting data on errors that occurred during transmission by various departments in history, analyzing the data and extracting characteristic factors that caused the errors;

[0066] S200, calculating a threshold value for determining whether an error occurs in data transmission based on the characteristic factor values ​​of each department in the history of data transmission;

[0067] S300, extracting historical data records of errors in data transmission by various departments, analyzing the relationship between all data and characteristic factors, finding credible parameters that have an influencing relationship with the characteristic factors, and calculating the credibility threshold of data error warning judgment;

[0068] S400, for the real-time transmitted data, using the error threshold to determine whether the real-time data transmitted by each department has an error, and issuing an error warning in real time;

[0069] S500: Calculate the credibility of the error warning issued in real time, and determine whether the error warning is credible according to the credibility threshold.

[0070] In step S100, the specific steps of extracting the characteristic factors causing errors after analyzing the data are as follows:

[0071] S101, collect data {Q1, Q2, Q3, ..., Qn} of errors in transmission of each department in history; Q1, Q2, Q3, ..., Qn represent the 1st, 2nd, 3rd, ..., nth error data values ​​in transmission of each department in history, and n is a positive integer; collect data of errors in transmission of each department in history m times; and collect data of no errors in transmission n times, and collect m times in the same way;

[0072] S102, collect n data of errors and no errors in the transmission of each department in the m times of history, and calculate the average value of the change when each department transmits data, the formula is:

[0073]

[0074] Among them, Q(p) represents the average value of the change when each department transmits data, Q' represents the data value without error when each department transmits data, Q represents the data value with error when each department transmits data, n represents the number of collected data, and M represents the number of influencing factors of data error; the influencing factors include data transmission delay, data update frequency and data recording time deviation, etc.

[0075] S103. Calculate the average value of the change in transmission data of each department as {Q(p1), Q(p2), Q(p3), ..., Q(pl)}; Q(p1), Q(p2), Q(p3), ..., Q(pl) represent the calculated average values ​​of the change in transmission data of the 1st, 2nd, 3rd ... lth departments, conduct comparative analysis on the average values ​​of the change in l departments, extract the departments whose average values ​​of transmission data change are greater than 50%, count the influencing factors that cause data errors, and record the influencing factors with the largest number as characteristic factors; check the characteristic factor values ​​T(Qk) of each department when transmitting data, where k represents the characteristic labels of different characteristic factors.

[0076] In step S200, the specific steps of calculating the threshold value for determining the error in data transmission of each department are:

[0077] S201, collect the characteristic factor values ​​of each department in the history of data transmission {{T(Q1 k1 ), T(Q1 k2 ), T(Q1 k3 ), ..., T(Q1 kj )), {T(Q2 k1 ), T(Q2 k2 ), T(Q2 k3 ), ..., T(Q2 kj )), ..., {T(Qm k1 ), T(Qm k2 ), T(Qm k3 ), ..., T(Qm kj )}}; T(Q1), T(Q2), T(Q3), ..., T(Qm) represent the characteristic factor values ​​of the collected transmission data when an error occurs for the 1st, 2nd, 3rd, ..., mth time, m is a positive integer, and {k1, k2, k3, ..., kj} represent the characteristic labels of different characteristic factors respectively;

[0078] S202. Calculate the error threshold according to the characteristic factor values ​​of errors that occurred in data transmission in each department in history. The formula is:

[0079]

[0080] Among them, r represents the error threshold for judging the error in data transmission of each department, T(Qu kv ) represents the characteristic factor value of the collected transmission data when an error occurs.

[0081] In step S300, the specific steps of searching for a credible parameter that has an influencing relationship with a characteristic factor are:

[0082] S311. Collect K influencing factors (current network speed and department data update frequency, etc.) when characteristic factors change during data transmission in each department in history, draw a linear graph with the influencing factors as independent variables and the characteristic factors as dependent variables, observe and analyze the drawn linear graph, establish a regression model for each influencing factor and characteristic factor, and use the regression model to represent the relationship between each influencing factor and characteristic factor; the calculation expression is:

[0083] T”(Q k )=a×h+b

[0084] Among them, T”(Q k ) represents the predicted value of the characteristic factor, h represents the value of the influencing factor, a represents the slope of the regression model, and b represents the intercept of the regression model;

[0085] S312. Calculate the slope and intercept of the regression model. The specific formula is:

[0086]

[0087] b=T′(Q f )-a×h′(f)

[0088] Among them, e represents the number of collected influencing factors and each characteristic factor, h(f) represents the value of each influencing factor, T(Q f ) represents the different characteristic factor values ​​collected, h'(f) represents the average value of the influencing factor, T'(Qf) represents the average value of the characteristic factor, f∈[1,e]; the expressions of the regression model of the K characteristic factors are calculated in turn, and the predicted values ​​of the K characteristic factors are finally obtained as {T"(Q k )_1,T”(Q k )_2,T”(Q k )_3, ..., T”(Q k )_K};T”(Q k )_1,T”(Q k )_2,T”(Q k )_3, ..., T”(Q k )_K represent the predicted values ​​of the 1st, 2nd, 3rd, ..., Kth characteristic factors respectively;

[0089] S313. After calculating the predicted value of each characteristic factor, the difference between the predicted value and the true value of each characteristic factor is calculated. The formula is:

[0090]

[0091] Among them, ch represents the difference between the predicted value and the true value of the characteristic factor in each regression model, T(Q k) represents the true value of the characteristic factor of each point, T”(Q k ) represents the predicted value of the characteristic factor, and e represents the number of characteristic factor values; after calculation, the K difference values ​​are {ch_1, ch_2, ch_3, ..., ch_K}, where ch_1, ch_2, ch_3, ..., ch_K represent the calculated difference values ​​between the 1st, 2nd, 3rd, ..., Kth characteristic factor predicted values ​​and the true values, respectively;

[0092] S314: Calculate the median value of the K difference values, specifically in the following manner:

[0093] Sort ch_1, ch_2, ch_3, ..., ch_K in descending order, and take Differences is the median value; when When , it is judged that the characteristic factor and the influencing factor corresponding to the difference value ch do not have an influence relationship. When , it is judged that the characteristic factor corresponding to the difference value ch and the influencing factor have an influence relationship, and the influencing factor with an influence relationship is defined as the credible factor B.

[0094] A regression model is constructed for each type of environmental data. The difference value of each regression model is calculated to determine whether there is an influence relationship between the corresponding influencing factor and the characteristic factor, and to find out the credible parameters in the influencing factor that have an influence relationship with the characteristic factor. The credible parameters can be used to verify whether the early warning judgment is credible, which greatly improves the credibility of the system's early warning judgment and reduces the probability of judgment errors.

[0095] In step S300, the specific steps of calculating the credibility threshold of data error warning judgment are:

[0096] S321. The trusted parameters affected by the calculated and characteristic factors are {B1, B2, B3, ..., Bd}, where B1, B2, B3, ..., Bd represent the 1st, 2nd, 3rd, ..., dth trusted parameters obtained by calculation, and d is a positive integer; collect the values ​​of each trusted parameter in which errors occurred in the data of each department in history;

[0097] S322. Calculate the credibility threshold of the error warning according to the credible parameter value, and the formula is:

[0098]

[0099] Among them, re represents the credibility threshold of data warning when data errors occur, B represents the trusted parameter value when each department transmits data, and d represents the number of trusted parameters.

[0100] In step S400, the error threshold is used to determine whether an error occurs in the real-time data transmitted by each department:

[0101] For the characteristic factor value T(D) of real-time transmitted data, use the error threshold calculated in S200 to judge the real-time characteristic factor value. When T(D) ≥ r, it is judged that the real-time transmitted data has an error and an error warning is issued. When T(D) < r, it is judged that the real-time transmitted data is normal.

[0102] In step S500, the specific steps for calculating the credibility of the real-time error warning issued are as follows:

[0103] S501. After judging the real-time transmitted data and issuing a warning, extract the regression model of each credibility parameter and the characteristic factor in S312, and calculate all real-time credibility parameter values using the real-time characteristic factor value T(D). Specifically:

[0104] Bs = a × T(D) + b

[0105] Calculate each real-time credibility parameter value as {Bs1, Bs2, Bs3,..., Bsd}, where Bs1, Bs2, Bs3,..., Bsd respectively represent the 1st, 2nd, 3rd,..., dth real-time credibility parameter values;

[0106] S502. Calculate the credibility of the error warning according to the real-time credibility parameter values. The formula is:

[0107]

[0108] In the formula, re′ represents the credibility of the real-time calculated error warning, Bs represents each real-time calculated credibility parameter value, and d represents the number of credibility parameters;

[0109] S503. Extract the credibility threshold calculated in S322 to judge the credibility of the real-time error warning. When re′ < re, it is judged that the credibility of the error warning is not high and no warning operation is performed; when re′ ≥ re, it is judged that the error warning is credible, a data anomaly warning is issued, and the data source is immediately checked and the data is retransmitted.

[0110] A data supervision system based on a unified data exchange service, which includes a historical data collection and error analysis module, an error threshold calculation module, a characteristic relationship analysis module, a real-time transmission monitoring and error warning module, and a warning credibility calculation and management module;

[0111] The historical data collection and error analysis module is responsible for collecting the historical data of errors that occur during the transmission of each department, cleaning and analyzing these data, and extracting the characteristic factors that cause transmission errors;

[0112] The error threshold calculation module is used to calculate a threshold for determining whether an error occurs in real-time data transmission based on characteristic factors extracted from historical data;

[0113] The characteristic relationship analysis module is used to extract data records of transmission errors in history, analyze the relationship between these data and characteristic factors, determine the credible parameters that affect the transmission errors, and calculate the credibility threshold of the error warning;

[0114] The real-time transmission monitoring and error warning module is used to monitor the real-time data transmission of each department, and evaluate whether the data has error risks in real time according to the calculated error threshold, and issue error warnings in time;

[0115] The warning credibility calculation and management module is used to calculate the credibility of the real-time error warning and judge the reliability of the warning based on the credibility threshold; if the credibility of the warning exceeds the threshold, the warning is considered to be credible and further processed.

[0116] The historical data collection and error analysis module includes a historical data collection unit, a metadata management unit, a data exchange unit and an error analysis unit;

[0117] The data collection unit is used to collect data records transmitted by various departments in history; the metadata management unit is used to record the source department of the data, the sending and receiving time and the data type, and to track the source of the data and resend it after the system issues an error warning; the data exchange unit is used to use the external network-intranet cross method to exchange data, and the internal network is used for data transmission between each department and the supervision system. After the supervision system makes a decision on the received data, it uses the external network to send the decision to the destination departments; the error analysis unit is used to extract key factors in historical data and then perform error analysis on the data.

[0118] In an embodiment of the present invention,

[0119] Data collection: From historical data, we collected data sets of errors and data sets without errors in the data transmission process of four departments A, B, C, and D. The details are as follows:

[0120] Department A: Error data set {QA1, QA2, QA3, ..., QA10}, normal data set {QA'1, QA'2, QA'3, ..., QA'10}.

[0121] Department B: Error data set {QB1, QB2, QB3, ..., QB10}, normal data set {QB'1, QB'2, QB'3, ..., QB'10}.

[0122] Department C: Error data set {QC1, QC2, QC3, ..., QC10}, normal data set {QC'1, QC'2, QC'3, ..., QC'10}.

[0123] Department D: Error data set {QD1, QD2, QD3, ..., QD10}, normal data set {QD'1, QD'2, QD'3, ..., QD'10}.

[0124] Calculate the average value of the change in each department's transmission data, the formula is:

[0125]

[0126] Among them, Q(p) represents the average value of the change when each department transmits data, Q' represents the data value without error when each department transmits data, Q represents the data value with error when each department transmits data, n represents the number of data collected, and M represents the number of factors affecting data errors;

[0127] Q(pA)=0.72*100%=72%,

[0128] Q(pB)=0.45*100%=45%,

[0129] Q(pC)=0.55*100%=55%,

[0130] Q(pD)=0.67*100%=67%;

[0131] Comparative analysis shows that the average value of transmission data changes in departments A, C, and D exceeds 50%, so the main influencing factors of these departments are extracted as characteristic factors.

[0132] Characteristic factors of department A: transmission delay (T(QA) = transmission delay);

[0133] Characteristic factors of department C: data update frequency (T(QC) = data update frequency);

[0134] Characteristic factors of department D: data recording time deviation (T(QD) = data recording time deviation);

[0135] Step S200: Calculation of error judgment threshold

[0136] Characteristic factor value collection:

[0137] Collect the characteristic factor value set of each department in the historical data, as follows:

[0138] Department A: {T(QA1), T(QA2), T(QA3), ..., T(QA10)};

[0139] Department B: {T(QB1), T(QB2), T(QB3), ..., T(QB10)};

[0140] Department C: {T(QC1), T(QC2), T(QC3), ..., T(QC10)};

[0141] Department D: {T(QD1), T(QD2), T(QD3), ..., T(QD10)};

[0142] Calculate the error threshold using the formula:

[0143]

[0144] Among them, r represents the error threshold for judging the error in data transmission of each department, T(Qu kv ) represents the characteristic factor value of the collected transmission data when an error occurs.

[0145] Error threshold for department A: r(A) = 0.65;

[0146] Error threshold for department B: r(B) = 0.50;

[0147] Error threshold for department C: r(C) = 0.55;

[0148] Error threshold of department D: r(D) = 0.62;

[0149] Linear regression model establishment:

[0150] The data of each department are analyzed and a linear regression model is established between each influencing factor and characteristic factor.

[0151] Department A (characteristic factor: transmission delay):

[0152] T″(QA)=1.2×h+0.3

[0153] Department C (characteristic factor: data update frequency):

[0154] T″(QB)=0.8×h+0.4

[0155] Department D (characteristic factor: data recording time deviation):

[0156] T″(QC)=1.5×h+0.2

[0157] Through regression analysis, a set of credible parameters is obtained.

[0158] Department A's trusted parameters: B1 = network speed, B2 = network jitter;

[0159] Department C trusted parameters: B1 = packet size, B2 = transmission frequency;

[0160] Department D trusted parameters: B1 = timestamp accuracy, B2 = server response time;

[0161] Credible parameter difference calculation:

[0162] Calculate the regression model difference value between the credible parameters and characteristic factors of each department, sort and extract the credible factors.

[0163] The difference value of department A is ch_A = {0.10, 0.15}, and the credible parameters are B1 and B2;

[0164] The difference value of department C is ch_C = {0.05, 0.12}, and the credible parameters are B1 and B2;

[0165] The difference value of department D is cn_D = {0.08, 0.14}, and the credible parameters are B1 and B2;

[0166] Real-time data analysis:

[0167] Analyze the characteristic factors of the real-time transmitted data to determine whether an error has occurred:

[0168] The real-time characteristic factor value of department A is T(DA)=0.70, which satisfies T(DA)≥rA(0.65), triggering an error warning.

[0169] Calculate real-time trusted parameters:

[0170] Use regression models to analyze real-time data and calculate real-time trustworthy parameters:

[0171] Real-time trustworthy parameters of department A: Bs_A = {0.68, 0.70};

[0172] The formula for calculating the credibility of the false warning is:

[0173]

[0174] In the formula, re′ represents the credibility of the error warning calculated in real time, Bs represents each credible parameter value calculated in real time, and d represents the number of credible parameters;

[0175] The real-time warning credibility re'_A=0.69 calculated by Department A satisfies re'_A≥re_A(0.65), so a warning is issued and it is recommended to retransmit the data.

[0176] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0177] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data supervision method based on a unified data exchange service, characterized in that: The method comprises the following steps: S100, collecting data on errors that occurred during transmission by various departments in history, analyzing the data and extracting characteristic factors that caused the errors; S200, calculating an error threshold for determining whether an error occurs in the data transmission according to the characteristic factor values ​​of each department in the history of data transmission; S300, extracting historical data records of errors in data transmission by various departments, analyzing the relationship between all data and characteristic factors, finding credible parameters that have an influencing relationship with the characteristic factors, and calculating the credibility threshold of data error warning judgment; S400, for the real-time transmitted data, using the error threshold to determine whether the real-time data transmitted by each department has an error, and issuing an error warning in real time; S500 , calculating the credibility of the error warning issued in real time, and judging whether the error warning is credible according to the credibility threshold.

2. The data supervision method based on unified data exchange service according to claim 1, characterized in that: In step S100, the specific steps of extracting the characteristic factors causing errors after analyzing the data are as follows: S101, collect the data {Q1, Q2, Q3, ..., Qn} of errors in transmission of each department in history; Q1, Q2, Q3, ..., Qn represent the 1st, 2nd, 3rd, ..., nth error data values ​​in transmission of each department in history, n is a positive integer; collect the data of errors in transmission of each department in m times of history; And collect data without error during n transmissions, and do the same m times; S102, collect n data of errors and no errors in the transmission of each department in the m times of history, and calculate the average value of the change when each department transmits data, the formula is: Among them, Q(p) represents the average value of the change when each department transmits data; Q' represents the data value without error when each department transmits data; Q represents the data value with error when each department transmits data; n represents the number of data collected, and M represents the number of factors affecting the data error; S103, calculate the average value of the change of the transmission data of each department as {Q(p1), Q(p2), Q(p3), ..., Q(pl)}; Q(p1), Q(p2), Q(p3), ..., Q(pl) represent the calculated average values ​​of the transmission data of the 1st, 2nd, 3rd ... lth departments, compare and analyze the average values ​​of the changes of the l departments, extract the departments whose transmission data change average values ​​are greater than 50%, count the influencing factors that cause data errors, and record the influencing factors with the largest number as characteristic factors; check the characteristic factor values ​​T(Q k ), k represents the feature labels of different feature factors.

3. The data supervision method based on unified data exchange service according to claim 2 is characterized in that: In step S200, the specific steps of calculating the threshold value for determining the error in data transmission of each department are: S201, collect the characteristic factor values ​​of each department in the history of data transmission {{T(Q1 k1 ), T(Q1 k2 ), T(Q1 k3 ), ..., T(Q1 kj )},{T(Q2 k1 ), T(Q2 k2 ), T(Q2 k3 ), ..., T(Q2 kj )}, ..., {T(Qm k1 ), T(Qm k2 ), T(Qm k3 ), ..., T(Qm kj )}}; T(Q1), T(Q2), T(Q3), ..., T(Qm) represent the characteristic factor values ​​of the collected transmission data when an error occurs for the 1st, 2nd, 3rd ... mth time, m is a positive integer, { k1 , k2 , k3 , ..., kj } respectively represent the feature labels of different feature factors; S202. Calculate the error threshold according to the characteristic factor values ​​of errors that occurred in data transmission in each department in history. The formula is: Among them, r represents the error threshold for judging the error in data transmission of each department, T(Qu kv ) represents the characteristic factor value of the collected transmission data when an error occurs.

4. The data supervision method based on unified data exchange service according to claim 3 is characterized in that: In step S300, the specific steps of searching for a credible parameter that has an influencing relationship with a characteristic factor are: S311. Collect K influencing factors when characteristic factors change during data transmission in various departments in history, draw a linear graph with the influencing factors as independent variables and the characteristic factors as dependent variables, observe and analyze the drawn linear graph, establish a regression model for each influencing factor and characteristic factor, and use the regression model to represent the relationship between each influencing factor and characteristic factor; the calculation expression is: T″(Q k )=a×h+b Among them, T″(Q k ) represents the predicted value of the characteristic factor, h represents the value of the influencing factor, a represents the slope of the regression model, and b represents the intercept of the regression model; S312. Calculate the slope and intercept of the regression model. The specific formula is: b=T′(Q f )-a×h′(f) Where e represents the number of impact factors and characteristic factors collected; h(f) represents the value of each impact factor; T(Q f ) represents the collected characteristic factor value; h'(f) represents the average value of the influencing factor; T'(Q f ) represents the average value of the characteristic factor, f∈[1,e]; the expressions of the regression model of the K characteristic factors are calculated in turn, and the predicted value of the K characteristic factors is finally obtained as {T″(Q k )_1,T″(Q k )_2,T″(Q k )_3,...,T″(Q k )_K};T″(Q k )_1,T″(Q k )_2,T″(Q k )_3,...,T″(Q k )_K represent the predicted values ​​of the 1st, 2nd, 3rd, ..., Kth characteristic factors respectively; S313. After calculating the predicted value of each characteristic factor, the difference between the predicted value and the true value of each characteristic factor is calculated. The formula is: Among them, ch represents the difference between the predicted value and the true value of the characteristic factor in each regression model; T(Q k ) represents the true value of the characteristic factor of each point; T”(Q k ) represents the predicted value of the characteristic factor; e represents the number of characteristic factor values; after calculation, the K difference values ​​are {ch_1, ch_2, ch_3, ..., ch_K}, ch_1, ch_2, ch_3, ..., ch_K represent the calculated difference values ​​between the 1st, 2nd, 3rd, ..., Kth characteristic factor predicted values ​​and the true values ​​respectively; S314: Calculate the median value of the K difference values, specifically in the following manner: Sort ch_1, ch_2, ch_3, ..., ch_K in descending order, and take Differences is the median value; when When , it is judged that the characteristic factor and the influencing factor corresponding to the difference value ch do not have an influence relationship. When , it is judged that the characteristic factor corresponding to the difference value ch and the influencing factor have an influence relationship, and the influencing factor with an influence relationship is defined as the credible factor B.

5. The data supervision method based on unified data exchange service according to claim 4 is characterized in that: In step S300, the specific steps of calculating the credibility threshold of data error warning judgment are: S321. The credible parameters affected by calculation and characteristic factors are {B1, B2, B3,..., Bd}, where B1, B2, B3,..., Bd represent the 1st, 2nd, 3rd,..., dth credible parameters obtained by calculation, and d is a positive integer; collect each credible parameter value of data errors occurred in each department in history. S322. Calculate the credibility threshold of error warning according to the credible parameter value, and the formula is: where re represents the credibility threshold of data warning when data error occurs, B represents the credible parameter value when each department transmits data, and d represents the number of credible parameters.

6. The data supervision method based on unified data exchange service according to claim 5 is characterized in that: In step S400, the specific method for determining whether the real-time data transmitted by each department has an error by using the error threshold is as follows: For the characteristic factor value T(D) of real-time transmitted data, use the error threshold calculated in S200 to judge the real-time characteristic factor value. When T(D)≥r, it is judged that the real-time transmitted data has an error and an error warning is issued. When T(D)<r, it is judged that the real-time transmitted data is normal.

7. The data supervision method based on unified data exchange service according to claim 6 is characterized in that: In step S500, the specific steps for calculating the credibility of the real-time issued error warning are as follows: S501. After judging the real-time transmitted data and issuing a warning, extract the regression model of each credible parameter and characteristic factor in S312, and calculate all real-time credible parameter values by using the real-time characteristic factor value T(D). Specifically: Bs = a×T(D)+b Calculate each real-time credible parameter value as {Bs1, Bs2, Bs3,..., Bsd}, where Bs1, Bs2, Bs3,..., Bsd respectively represent the 1st, 2nd, 3rd,..., dth credible parameter values in real time. S502. Calculate the credibility of the error warning according to the real-time credible parameter value, and the formula is: where re’ represents the credibility of the error warning calculated in real time, Bs represents each real-time calculated credible parameter value, and d represents the number of credible parameters. S503. Extract the credibility threshold calculated in S322 to judge the credibility of the real-time error warning. When re’<re, it is judged that the credibility of the error warning is not high and no warning operation is performed; when re’≥re, it is judged that the error warning is credible, a data anomaly warning is issued, and the data source is immediately checked and the data is retransmitted.

8. A data supervision system based on a unified data exchange service, characterized by: The system includes a historical data collection and error analysis module, an error threshold calculation module, a characteristic relationship analysis module, a real-time transmission monitoring and error warning module, and a warning credibility calculation and management module. The historical data collection and error analysis module is responsible for collecting the historical data of errors occurred in the transmission process of each department, cleaning and analyzing these data, and extracting the characteristic factors leading to transmission errors. The error threshold calculation module is used to calculate the error threshold for judging whether real-time data transmission has an error according to the characteristic factors extracted from historical data. The characteristic relationship analysis module is used to extract the data records of transmission errors in history, analyze the relationship between these data and characteristic factors, determine the credible parameters affecting transmission errors, and calculate the credibility threshold of error warning. The real-time transmission monitoring and error warning module is used to monitor the real-time data transmission of each department, and evaluate whether the data has error risks in real time according to the calculated error threshold, and issue error warnings in time; The warning credibility calculation and management module is used to calculate the credibility of the real-time error warning and determine the reliability of the warning based on the credibility threshold; If the credibility of the warning exceeds the threshold, the warning is considered credible and further processed.

9. The data supervision system based on unified data exchange service according to claim 8, characterized in that: The historical data collection and error analysis module includes a historical data collection unit, a metadata management unit, a data exchange unit and an error analysis unit; The data collection unit is used to collect data records transmitted by various departments in history; the metadata management unit is used to record the source department of the data, the sending and receiving time and the data type, and to trace the source and resend the data after the system issues an error warning; the data exchange unit is used to exchange data using an external network-intranet crossover method, and the departments and the supervision system use the intranet for data transmission. After the supervision system makes a decision on the received data, it uses the external network to send the decision to the destination departments; the error analysis unit is used to extract key factors in historical data and then perform error analysis on the data.

Images