Fault data source determination method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of big data analysis, and discloses a fault data source determination method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring a first data source at the current moment, a second data source at the current moment and associated content information; determining a first target data source corresponding to the first data source at the current moment according to the correlation coefficient, the cross correlation coefficient and the first data source at the current moment; wherein the first target data source is an expected data source of the second data source at the current moment; if the residual value between the target data source and the second data source is greater than a residual threshold, determining target area information; wherein the target area information is used for indicating target areas applied by the first data source at the current moment and the second data source at the current moment; and determining a second target data source from the first data source at the current moment and the second data source at the current moment according to the target area information.

Description

Technical Field

[0001] The present invention relates to the technical field of big data analysis, and in particular to a method, device, computer equipment, storage medium and program product for determining a fault data source. Background Art

[0002] For large-scale distributed systems, large-scale support systems are often built to aggregate and collect business indicators for billing or operation and maintenance needs. C ontent In a CDN, both operators and customers need to understand the traffic generated by customer services in real time. This requires collecting data from all nodes in the entire CDN network, converging and calculating in the support system, and then outputting the calculated data source.

[0003] At present, three data sources are often used, which are the main data source and two auxiliary data sources; one auxiliary data source is used to provide backup data when the main data source has problems, and the other auxiliary data source is a data source that is not completely the same as the first two data sources, which is used to further increase the diversity and reliability of data. The data provided by the three data sources are compared in real time to check the consistency and differences between the data. According to the comparison results, decision logic is formulated to determine which data source is accurate and reliable, and then the reliable data source is output.

[0004] However, more data sources require more hardware resources to store and process the data, which increases costs. Summary of the invention

[0005] In view of this, the present invention provides a method, apparatus, computer equipment, storage medium and program product for determining a fault data source.

[0006] In a first aspect, the present invention provides a method for determining a faulty data source, obtaining a first data source at a current moment, a second data source at a current moment, and associated content information; wherein the associated content information includes: a residual threshold, a correlation coefficient, and a mutual correlation coefficient; determining a first target data source corresponding to the first data source at the current moment according to the correlation coefficient, the mutual correlation coefficient, and the first data source at the current moment; wherein the first target data source is an expected data source of the second data source at the current moment; if the residual value between the target data source and the second data source is greater than the residual threshold, determining target area information; wherein the target area information is used to indicate a target area to which the first data source at the current moment and the second data source at the current moment are applied; determining a second target data source from the first data source at the current moment and the second data source at the current moment according to the target area information; wherein the second target data source is a data source between the first data source at the current moment and the second data source at the current moment where no fault has occurred.

[0007] The method for determining a faulty data source provided in this embodiment can deeply understand the degree of association and mutual influence between data sources by introducing associated content information (including residual threshold, correlation coefficient and mutual correlation coefficient). According to the correlation coefficient, mutual correlation coefficient and the first data source at the current moment, the first target data source corresponding to the first data source at the current moment is determined, and if the residual value between the target data source and the second data source is greater than the residual threshold, the target area information is determined, thereby obtaining the area to which the first data source at the current moment and the second data source at the current moment are applied, and according to the first data source at the current moment and the second data source at the current moment of all areas, the second target data source is determined from the first data source at the current moment and the second data source at the current moment. That is, there is no need to introduce other data sources, and the data source that has not failed can be determined only through the first data source at the current moment and the second data source at the current moment, thereby reducing costs.

[0008] In one possible implementation, according to the target area information, a second target data source is determined from the first data source at the current moment and the second data source at the current moment, including: according to the target area, determining the first vector dot product between the first data source at the current moment and the first data source at the moment before the current moment; according to the target area, determining the second vector dot product between the second data source at the current moment and the second data source at the moment before the current moment; comparing the first vector dot product and the second vector dot product to generate a comparison result; wherein, when the comparison result is that the first vector dot product is greater than the second vector dot product, the second data source at the current moment is used as the second target data source, and when the comparison result is that the first vector dot product is less than the second vector dot product, the first data source at the current moment is used as the second target data source.

[0009] The method for determining the faulty data source provided in this embodiment can capture the dynamic changes of data in real time by calculating the vector dot product between the data sources at the current moment and the previous moment. When the vector dot product between the data sources changes significantly, it usually means that the change trend between the data sources has changed significantly. Then, by comparing the first vector dot product and the second vector dot product, it can be accurately determined which data source has a greater change trend between the first data source at the current moment and the second data source at the current moment, and the data source with a greater change trend is used as the faulty data source, thereby improving the accuracy of identifying the faulty data source.

[0010] In one possible implementation, if the residual value between the target data source and the second data source is greater than the residual threshold, the target area information is determined, including: if the residual value between the target data source and the second data source is greater than the residual threshold, detecting whether a first internal data source corresponding to the first data source at the current moment and a second internal data source corresponding to the second data source at the current moment are the same; if the first internal data source and the second internal data source are the same, determining the target area information.

[0011] The method for determining the faulty data source provided in this embodiment, if the residual value between the target data source and the second data source is greater than the residual threshold, firstly, by judging whether the first internal data source corresponding to the first data source at the current moment generated by the internal business and the second internal data source corresponding to the second data source at the current moment are the same, it is possible to quickly locate whether the problem originates from the internal data processing or transmission link; if the first internal data source and the second internal data source are the same, it indicates that there is no problem with the internal data source, and the target area information is determined, avoiding blindly looking for problems in external data sources or more complex systems, thereby greatly improving the diagnostic efficiency of the faulty data source.

[0012] In one possible implementation, the method further includes: if the first internal data source and the second internal data source are not the same, obtaining the first internal data source and the second internal data source at the adjacent moment of the current moment; detecting whether the first internal data source at the adjacent moment of the current moment and the first internal data source at the current moment are the same; detecting whether the second internal data source at the adjacent moment of the current moment and the second internal data source at the current moment are the same; if the first internal data source at the adjacent moment of the current moment and the first internal data source at the current moment are the same, using the second data source at the current moment as the second target data source; if the second internal data source at the adjacent moment of the current moment and the second internal data source at the current moment are the same, using the first data source at the current moment as the second target data source.

[0013] The method for determining the faulty data source provided in this embodiment can further verify the accuracy and consistency of the data of the data source by comparing the internal data source at the current moment and the adjacent moment. If the internal data source at the adjacent moment is the same as that at the current moment, it means that the data of the data source is continuous in time. If the internal data source at the adjacent moment is different from that at the current moment, there is a large fluctuation in the data of the data source, and there is a problem with the internal business data source. When there is a problem with the internal business data source, by quickly comparing and verifying the internal data source, the decision time is shortened, and it can be determined more quickly which data source is more suitable as the second target data source, thereby improving decision efficiency.

[0014] In one possible implementation, the process of determining associated content information includes: obtaining a first data source and a second data source at a historical moment; determining a correlation coefficient and a residual threshold by least squares fitting based on the first data source and the second data source at the historical moment; obtaining the first data source and the second data source at a target moment in the historical moment; wherein each time period within the target moment includes a data source; and determining a correlation coefficient based on data points in each time period.

[0015] The method for determining the fault data source provided in this embodiment can accurately determine the correlation coefficient between the first data source and the second data source through least squares fitting, thereby determining the correlation and change trend between the two data sources. At the same time, the setting of the residual threshold can accurately identify whether there is a fault in the system.

[0016] At the same time, the mutual correlation coefficient is an important indicator to measure the degree of linear correlation between two sets of data. By calculating the mutual correlation coefficient between the data points in each time period within the target time, the correlation between the first data source and the second data source can be accurately evaluated.

[0017] In a possible implementation, the method further includes: reporting a third target data source; wherein the third target data source is a data source that has not failed between the first data source at the current moment and the second data source at the current moment.

[0018] The method for determining a faulty data source provided in this embodiment displays the data sources that have not failed between the first data source at the current moment and the second data source at the current moment, so that the user can intuitively see which data source is reliable and has not failed at the current moment. This helps the user to quickly understand the operating status and data quality of the system.

[0019] In a second aspect, the present invention provides a device for determining a faulty data source, the device comprising: an acquisition module, used to acquire a first data source at a current moment, a second data source at a current moment, and associated content information; wherein the associated content information comprises: a residual threshold, a correlation coefficient, and a mutual correlation coefficient; a first determination module, used to determine a first target data source corresponding to the first data source at a current moment according to the correlation coefficient, the mutual correlation coefficient, and the first data source at a current moment; wherein the first target data source is an expected data source of the second data source at the current moment; a second determination module, used to determine target area information if the residual value between the target data source and the second data source is greater than the residual threshold; wherein the target area information is used to indicate a target area to which the first data source at a current moment and the second data source at a current moment are applied; a third determination module, used to determine a second target data source from the first data source at a current moment and the second data source at a current moment according to the target area information; wherein the second target data source is a data source in which a fault occurs between the first data source at a current moment and the second data source at a current moment.

[0020] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the method for determining the source of a fault data in the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0021] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method for determining a fault data source of the first aspect or any corresponding embodiment thereof.

[0022] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the method for determining a fault data source of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 is a flow chart of a method for determining a faulty data source according to an embodiment of the present invention;

[0025] Figure 2is a schematic diagram of a method for determining a fault data source according to an embodiment of the present invention;

[0026] Figure 3 is a structural block diagram of a device for determining a fault data source according to an embodiment of the present invention;

[0027] Figure 4 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0029] According to an embodiment of the present invention, an embodiment of a method for determining a faulty data source is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0030] In this embodiment, a method for determining a fault data source is provided, which can be used for computer equipment, such as a computer, a server, etc. Figure 1 FIG. 1 is a flow chart of a method for determining a faulty data source according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0031] Step S101, obtaining a first data source at a current moment, a second data source at a current moment, and associated content information; wherein the associated content information includes: a residual threshold, a correlation coefficient, and a mutual correlation coefficient.

[0032] The first data source can represent a data source indexed by business time (the time when an event occurs), which is often used as a gold standard for billing. The second data source can be used here to refer to a data source indexed by processing time, because most counter-based monitoring systems will discard the business time information of events. Among them, processing time can represent the time when the analysis program receives and processes an event. The current moment can represent the moment when the current service is triggered.

[0033] It should be noted that the first data source may be a data source for log monitoring, and the second data source may be a data source for a counter-based monitoring system.

[0034] The associated content information includes: residual threshold, correlation coefficient and mutual correlation coefficient. Among them, the residual refers to the difference between the two data sources, and the residual threshold is a preset threshold used to determine whether the difference between the two data sources is within an acceptable range. If the residual exceeds this threshold, it may mean that there is a problem with the system or a data source has failed. The correlation coefficient can be used to adjust the strength of the linear relationship between the two data sources, specifically to align the first data source and the second data source. The mutual correlation coefficient can be another indicator to measure the similarity between two data sources, but it focuses more on the correlation between two time series data. Unlike the correlation coefficient, the mutual correlation coefficient can capture the lagging or leading relationship between data.

[0035] Step S102, determining a first target data source corresponding to the first data source at the current moment according to the correlation coefficient, the mutual correlation coefficient, and the first data source at the current moment; wherein the first target data source is an expected data source of the second data source at the current moment.

[0036] The first target data source is the expected data source of the second data source at the current moment, that is, the second data source at the current moment is the real value, and the first target data source is the expected value. Specifically, when determining the first data source, the first target data source corresponding to the first data source at the current moment can be determined based on the correlation coefficient, the mutual correlation coefficient, and the first data source at the current moment.

[0037] Before making any prediction or estimation, it is usually necessary to collect historical data of the first data source and the second data source. These historical data are used to calculate the correlation coefficient and the mutual correlation coefficient. Using the historical data, the correlation coefficient and the mutual correlation coefficient between the first data source and the second data source are calculated by statistical methods. Based on the calculated correlation coefficient and the mutual correlation coefficient, and the value of the first data source at the current moment, a prediction model can be established, which can be used to predict the value of the second data source at the current moment, that is, the first target data source. The value of the first data source at the current moment is input into the prediction model to calculate the value of the first target data source.

[0038] As an example, determining the first target data source corresponding to the first data source at the current moment according to the correlation coefficient, the mutual correlation coefficient, and the first data source at the current moment includes:

[0039] l(t)=am(t+τ)+b; wherein l(t) is the first target data source, m(t) is the first data source at the current moment, a and b are both correlation coefficients, and τ is the mutual correlation coefficient.

[0040] Step S103, if the residual value between the target data source and the second data source is greater than the residual threshold, determine the target area information; wherein the target area information is used to indicate the target area applied by the first data source at the current moment and the second data source at the current moment.

[0041] After determining the first target data source and the second data source, it is necessary to determine whether the data source is faulty based on the residual value between the first target data source and the second data source. After the two data sources are aligned, it is possible to determine whether there is inconsistency between the data sources and decide on the data source that provides external services.

[0042] Specifically, in a data processing and monitoring system, when it is found that the residual value between the target data source (i.e., the expected value of the second data source predicted or calculated based on the first data source, the correlation coefficient and the mutual correlation coefficient) and the actual second data source exceeds a preset residual threshold, this indicates that there is a significant difference between the two data sources, which may mean that there is an abnormality in the system or a problem with a data source.

[0043] As an example, currently both data sources are operating normally, and the current predicted residual e(t) is:

[0044] e(t)=l(t)-l ′ (t)=l(t)-am(t+τ)-b; wherein l′(t) is the second data source at the current moment.

[0045] As an example, the residual follows the normal distribution e~N(0,σ 2 ) and calculate the residual sum of squares. When predicting, whether the residual is greater than 3σ can be used as a criterion for whether the dual data sources are inconsistent. That is, if e(t)>3σ, it is considered that the degree of inconsistency between the dual data sources is high enough, and the data source of the specific fault needs to be identified.

[0046] The target area may include the location of the data source, that is, when it is determined that a data source has a fault, it is necessary to determine the area where the data source has been used.

[0047] Step S104, determining a second target data source from the first data source at the current moment and the second data source at the current moment according to the target area information; wherein the second target data source is a data source where a fault occurs between the first data source at the current moment and the second data source at the current moment.

[0048] The second target data source is a data source that fails between the first data source at the current moment and the second data source at the current moment. Specifically, after determining the target area, the second target data source is determined from the first data source at the current moment and the second data source at the current moment according to data of all target areas.

[0049] As an example, the values ​​of the first data source and the second data source at the current moment can be analyzed to see whether they are within a reasonable range and whether they conform to the expected pattern or trend, and to check whether the data sources have obvious outliers, missing values, or inconsistencies, so as to determine the faulty data source.

[0050] As an example, a machine learning algorithm or a statistical method may be used to automatically detect faults in a data source, may be trained based on the characteristics of the data source, historical data, and fault patterns, and provide fault detection results in real time.

[0051] As an example, the faulty data source can be determined based on the comparison results between the first vector dot product between the first data source at the current moment and the first data source at the previous moment, and the second vector dot product between the second data source at the current moment and the second data source at the previous moment.

[0052] The method for determining a faulty data source provided in this embodiment can deeply understand the degree of association and mutual influence between data sources by introducing associated content information (including residual threshold, correlation coefficient and mutual correlation coefficient). According to the correlation coefficient, mutual correlation coefficient and the first data source at the current moment, the first target data source corresponding to the first data source at the current moment is determined, and if the residual value between the target data source and the second data source is greater than the residual threshold, the target area information is determined, thereby obtaining the area to which the first data source at the current moment and the second data source at the current moment are applied, and according to the first data source at the current moment and the second data source at the current moment of all areas, the second target data source is determined from the first data source at the current moment and the second data source at the current moment. That is, there is no need to introduce other data sources, and the data source that has not failed can be determined only through the first data source at the current moment and the second data source at the current moment, thereby reducing costs.

[0053] In a possible implementation, step S104 includes:

[0054] Step a1: determining, according to the target area, a first vector product between a first data source at a current moment and a first data source at a moment before the current moment.

[0055] Step a2: determining, according to the target area, a second vector product between the second data source at the current moment and the second data source at the moment before the current moment.

[0056] After determining the target area, the first vector dot product between the first data source at the current moment and the first data source at the previous moment is calculated. The value of the data source can be regarded as a series of points on a time series, and each point represents a data value at a moment. The vector dot product (dot product) is used in this context to measure the similarity or change trend between the data values ​​at two consecutive moments. Specifically, the first vector dot product can be determined by determining the product between the first data source vector at the current moment corresponding to the first data source at the current moment and the first data source vector at the previous moment corresponding to the first data source at the previous moment of the current moment.

[0057] As an example, the first data source vector A1 at the current moment may represent the value of the first data source at the current moment in the target region; the first data source vector A2 at the previous moment represents the value of the first data source at the previous moment before the current moment in the target region. The first vector dot product may be A1×A2.

[0058] Similarly, the method for determining the dot product of the second vector is the same as the method for determining the dot product of the first vector, and will not be described in detail here.

[0059] Step a3, compare the dot product of the first vector and the dot product of the second vector to generate a comparison result; wherein, when the comparison result is that the dot product of the first vector is greater than the dot product of the second vector, the second data source at the current moment is used as the second target data source; when the comparison result is that the dot product of the first vector is less than the dot product of the second vector, the first data source at the current moment is used as the second target data source.

[0060] After determining the first vector dot product and the second vector dot product, the data source with a smaller value can be taken as a fault data source by comparing the values ​​of the first vector dot product and the second vector dot product.

[0061] As an example, the regional data is included in the analysis. The data of the first data source and the second data source provided to the outside world both have regional dimensions, that is, they are:

[0062] l(t')=∑ r l r (t); m(t') = ∑ r m r (t); where r∈{1…N} is the region code, and there are N regions in total. The usage data at each moment can be regarded as an N-dimensional vector composed of regional data, and the difference change at different moments is measured by the vector angle at different moments, m r (t) is the second sub-data source corresponding to each area; l r (t) is the first sub-data source corresponding to each area.

[0063] In order to facilitate the calculation of vector angles and avoid the influence of magnitude on vector angles, the data of each region needs to be normalized first. Taking the usage of each region of log data l(t) as an example, we have:

[0064] After normalization, the vector dot product of the current moment and the previous moment can be calculated, which is the vector angle, representing the change in the regional distribution of the customer's business at the current moment and the previous moment. The vector dot product of the two data sources is calculated at the current moment and the previous moment.

[0065] That is: Among them, θ represents the vector angle. If the vector directions are closer, cos(θ) is closer to 1. If the vector directions are completely opposite, cos(θ) = -1. Therefore, to determine the data source with the smallest regional distribution change, we only need to determine the difference between cos(θ1) and cos(θ m ) can be of the same size. If cos(θ m )>cos(θ1), then at the current moment, the monitoring data source is used to provide services to the outside world; otherwise, the data of the log data source is provided to the outside world.

[0066] The method for determining the faulty data source provided in this embodiment can capture the dynamic changes of data in real time by calculating the vector dot product between the data sources at the current moment and the previous moment. When the vector dot product between the data sources changes significantly, it usually means that the change trend between the data sources has changed significantly. Then, by comparing the first vector dot product and the second vector dot product, it can be accurately determined which data source has a greater change trend between the first data source at the current moment and the second data source at the current moment, and the data source with a greater change trend is used as the faulty data source, thereby improving the accuracy of identifying the faulty data source.

[0067] In a possible implementation, step S103 includes:

[0068] Step b1: if the residual value between the target data source and the second data source is greater than the residual threshold, detect whether the first internal data source corresponding to the first data source at the current moment and the second internal data source corresponding to the second data source at the current moment are the same.

[0069] Step b2: if the first internal data source and the second internal data source are the same, determine the target area information.

[0070] As can be seen from the above, when it is found that the residual value between the first target data source and the actual second data source exceeds the preset residual threshold, this indicates that there is a significant difference between the two data sources, which may mean that there is an abnormality in the system or a problem with a data source. In addition, the customer's business requests will be counted as usage information and provided to the outside. In addition, internal requests will also be classified as internal business and their usage will be counted. For example, requests generated at fixed time intervals such as refresh pre-fetching, monitoring dialing, heartbeat information, etc. will be counted separately by the log and monitoring system and classified as internal business. If the first internal data source corresponding to the first data source at the current moment is the same as the second internal data source corresponding to the second data source at the current moment, it indicates that the internal dual data sources have not changed, and the difference in customer usage cannot be explained. There may be omissions or over-collection. In this case, it is necessary to determine the target area and determine the faulty data source from the customer business end based on the target area.

[0071] The method for determining the faulty data source provided in this embodiment, if the residual value between the target data source and the second data source is greater than the residual threshold, firstly, by judging whether the first internal data source corresponding to the first data source at the current moment generated by the internal business and the second internal data source corresponding to the second data source at the current moment are the same, it is possible to quickly locate whether the problem originates from the internal data processing or transmission link; if the first internal data source and the second internal data source are the same, it indicates that there is no problem with the internal data source, and the target area information is determined, avoiding blindly looking for problems in external data sources or more complex systems, thereby greatly improving the diagnostic efficiency of the faulty data source.

[0072] In a possible implementation, the method further includes:

[0073] Step c1: if the first internal data source and the second internal data source are different, obtain the first internal data source and the second internal data source at adjacent moments of the current moment.

[0074] The adjacent time of the current time can be used to represent the time point at which the first internal data source and the second internal data source are the same the most recently from the current time. Specifically, the first internal data source and the second internal data source of the adjacent time of the current time are determined by the current time.

[0075] As an example, determine an appropriate time window size based on the frequency of data updates and system requirements. For example, if the data is updated once a minute, the time window can be set to the data in the past few minutes. Retrieve the first internal data source and the second internal data source for all time points in the time window from the data store. Traverse the data in the time window and compare the first internal data source and the second internal data source at each time point. Record the first (i.e., the most recent) time point that satisfies the same first internal data source and the second internal data source. Once this time point is found, obtain the first internal data source and the second internal data source at that time point.

[0076] Step c2, detecting whether the first internal data source at the adjacent moment to the current moment is the same as the first internal data source at the current moment.

[0077] Step c3, detecting whether the second internal data source at the adjacent moment to the current moment is the same as the second internal data source at the current moment.

[0078] Under normal circumstances, since the internal requests are made at fixed time intervals and are evenly requested to all nodes, the first internal data source and the second internal data source are always the same. Therefore, it is possible to detect whether the first internal data source at the adjacent time of the current moment is the same as the first internal data source at the current moment, and whether the second internal data source at the adjacent time of the current moment is the same as the second internal data source at the current moment.

[0079] Step c4: if the first internal data source at the adjacent moment of the current moment is the same as the first internal data source at the current moment, the second data source at the current moment is used as the second target data source.

[0080] Step c5: If the second internal data source at the adjacent moment of the current moment is the same as the second internal data source at the current moment, the first data source at the current moment is used as the second target data source.

[0081] If the first internal data source at the adjacent moment of the current moment is the same as the first internal data source at the current moment, it indicates that the first data source at the current moment is not faulty and the second data source is a faulty data source. If the second internal data source at the adjacent moment of the current moment is the same as the second internal data source at the current moment, it indicates that the second data source at the current moment is not faulty and the first data source is a faulty data source.

[0082] As an example, the first internal data source is denoted as l internal (t) and the second internal data source is denoted as m internal (t). Under normal circumstances, since internal requests are made at fixed intervals and are evenly requested to all nodes, there is always l internal (t) = m internal(t); When a node is launched or the internal software version is updated, a short-term inconsistency will occur. Therefore, we can first determine whether l internal (t) = m internal (t), if not, then find the time point t that most recently satisfied the condition s , at this time, if l internal (t) = l internal (t s ) indicates that the log data is normal and l(t) is provided to the customer; if m internal (t) = m internal (t s ) indicates that the monitoring data is normal and m(t) is provided to the customer. At this time, it can be considered that the internal change caused the difference in the dual data sources for the customer's business.

[0083] The method for determining the faulty data source provided in this embodiment can further verify the accuracy and consistency of the data of the data source by comparing the internal data source at the current moment and the adjacent moment. If the internal data source at the adjacent moment is the same as that at the current moment, it means that the data of the data source is continuous in time. If the internal data source at the adjacent moment is different from that at the current moment, there is a large fluctuation in the data of the data source, and there is a problem with the internal business data source. When there is a problem with the internal business data source, by quickly comparing and verifying the internal data source, the decision time is shortened, and it can be determined more quickly which data source is more suitable as the second target data source, thereby improving decision efficiency.

[0084] In a possible implementation, the process of determining the associated content information includes:

[0085] Step d1, obtaining a first data source and a second data source at a historical moment.

[0086] The first data source and the second data source at the historical moment may be used to represent the first data source and the second data source one day or one month before the current moment.

[0087] Step d2, determining the correlation coefficient and the residual threshold value by least square fitting based on the first data source and the second data source at the historical moment.

[0088] According to the first data source and the second data source at the historical moment, the least squares fitting of the historical moment and the residual threshold corresponding to the residual distribution are calculated for observation and decision-making on the next day.

[0089] As an example, preprocess the historical data, such as denoising, standardization, etc. Use the least squares method to perform linear or nonlinear fitting on the first data source and the second data source to obtain the correlation coefficient (such as slope, intercept, etc.). Calculate the fitting residual, that is, the difference between the actual value and the fitted value. Set a reasonable residual threshold based on the residual distribution for subsequent data verification and anomaly detection.

[0090] Step d3, obtaining a first data source and a second data source of a target moment in the historical moments; wherein each time period in the target moment includes a data source.

[0091] The target time can be used to represent the data of the two hours before and after the peak usage time point in the historical moment. The target time is divided into multiple time periods (such as one hour, 5 minutes, etc.), and the data source is determined according to the time period, that is, each time period includes a data source.

[0092] Preferably, in order to ensure time resolution, it is necessary to use data with the same time granularity as that provided to the outside, such as statistical values ​​per minute. In order to enhance the effect of cross-correlation calculation, it is necessary to obtain data during the peak period of the previous day, such as data two hours before and after the peak time point of the previous day, with one data point per minute (one time period), for a total of 240 data points.

[0093] Step d4, determining the correlation coefficient based on the data points in each time period.

[0094] Perform cross-correlation analysis on the data points in each time period. Calculate the cross-correlation coefficient, which reflects the degree of linear correlation between two data sources. Analyze the change trend and stability of the cross-correlation coefficient to determine the synchronization and consistency between data sources.

[0095] As an example, the cross-correlation coefficient can be obtained by calculating the covariance of the first data source in the time period at the target moment and the second data source in the time period at the target moment at different delays, and dividing by the product of their respective standard deviations. The value range of the cross-correlation coefficient is usually between -1 and 1, where: a value close to 1 indicates a strong positive correlation between the two data sources. A value close to -1 indicates a strong negative correlation. A value close to 0 indicates that there is no obvious linear relationship between the two data sources. Then compare multiple cross-correlation coefficients to determine the cross-correlation coefficient to be used next.

[0096] The method for determining the fault data source provided in this embodiment can accurately determine the correlation coefficient between the first data source and the second data source through least squares fitting, thereby determining the correlation and change trend between the two data sources. At the same time, the setting of the residual threshold can accurately identify whether there is a fault in the system.

[0097] At the same time, the cross-correlation coefficient is an important indicator to measure the degree of linear correlation between two sets of data. By calculating the cross-correlation coefficient between the data points in each time period within the target time, the correlation between the first data source and the second data source can be accurately evaluated.

[0098] In a possible implementation, the method further includes: reporting a third target data source; wherein the third target data source is a data source that has not failed between the first data source at the current moment and the second data source at the current moment.

[0099] The third target data source is a data source that has not failed between the first data source at the current moment and the second data source at the current moment. After the third target data source is determined, the third target data source can be output to the user's terminal via email, FTP, API interface, cloud storage, etc., and then displayed.

[0100] In a possible implementation, the third target data source may be organized in a format required by the customer. This may include conversion of data types, reorganization of data fields, or standardization of data formats. The formatted and encrypted third target data source is packaged into a format acceptable to the customer, such as a file, a data packet, etc. The third target data source is output to the user's terminal via email, FTP, API interface, cloud storage, etc., and then displayed.

[0101] Please refer to Figure 2 , Figure 2 It is a schematic diagram of a method for determining a faulty data source provided according to an embodiment of the present invention.

[0102] Obtain the first data source at the current moment, the second data source at the current moment, and the associated content information; wherein the associated content information includes: residual threshold, correlation coefficient, and mutual correlation coefficient. Determine the residual value according to the correlation coefficient, mutual correlation coefficient, and the first data source at the current moment. Detect whether the residual value is greater than the residual threshold (such as 3ó); wherein, if the residual value is not greater than the residual threshold, report the log data. If the residual value is greater than the residual threshold, detect whether the first internal data source is the same as the second internal data source.

[0103] If the first internal data source and the second internal data source are not the same, detect whether the second internal data source is the same as the second internal data source at the adjacent time of the current time, and if the second internal data source is not the same as the second internal data source at the adjacent time of the current time, report the log data. If the second internal data source is the same as the second internal data source at the adjacent time of the current time, report the monitoring data.

[0104] If the first internal data source and the second internal data source are the same, detect whether the dot product of the first vector is greater than the dot product of the second vector. If the dot product of the first vector is less than the dot product of the second vector, report monitoring data; if the dot product of the first vector is greater than the dot product of the second vector, report log data.

[0105] The method for determining the faulty data source provided in this embodiment displays the data sources that have not failed between the first data source at the current moment and the second data source at the current moment, so that the user can intuitively see which data source is reliable and has not failed at the current moment. This helps the user quickly understand the operating status of the system and the data quality

[0106] In this embodiment, a device for determining a fault data source is also provided, and the device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made are not repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0107] This embodiment provides a device for determining a fault data source, such as Figure 3 As shown, it includes: an acquisition module 301, which is used to acquire the first data source at the current moment, the second data source at the current moment, and related content information; wherein the related content information includes: a residual threshold, a correlation coefficient, and a mutual correlation coefficient; a first determination module 302, which is used to determine the first target data source corresponding to the first data source at the current moment according to the correlation coefficient, the mutual correlation coefficient, and the first data source at the current moment; wherein the first target data source is the expected data source of the second data source at the current moment; a second determination module 303, which is used to determine the target area information if the residual value between the target data source and the second data source is greater than the residual threshold; wherein the target area information is used to indicate the target area to which the first data source at the current moment and the second data source at the current moment are applied; a third determination module 304, which is used to determine the second target data source from the first data source at the current moment and the second data source at the current moment according to the target area information; wherein the second target data source is a data source with a fault between the first data source at the current moment and the second data source at the current moment.

[0108] In one possible implementation, the third determination module 304 includes: a first determination unit, used to determine, based on the target area, a first vector dot product between a first data source at a current moment and a first data source at a moment before the current moment; a second determination unit, used to determine, based on the target area, a second vector dot product between a second data source at a current moment and a second data source at a moment before the current moment; a comparison unit, used to compare the first vector dot product and the second vector dot product to generate a comparison result; wherein, when the comparison result is that the first vector dot product is greater than the second vector dot product, the second data source at the current moment is used as the second target data source, and when the comparison result is that the first vector dot product is less than the second vector dot product, the first data source at the current moment is used as the second target data source.

[0109] In one possible implementation, the first determination module 302 includes: a detection unit, which is used to detect whether the first internal data source corresponding to the first data source at the current moment and the second internal data source corresponding to the second data source at the current moment are the same if the residual value between the target data source and the second data source is greater than the residual threshold; and a third determination unit, which is used to determine the target area information if the first internal data source and the second internal data source are the same.

[0110] In one possible implementation, the device further includes: an internal data source acquisition module, used to acquire the first internal data source and the second internal data source at an adjacent moment of the current moment if the first internal data source and the second internal data source are not the same; a first detection module, used to detect whether the first internal data source at an adjacent moment of the current moment is the same as the first internal data source at the current moment; a second detection module, used to detect whether the second internal data source at an adjacent moment of the current moment is the same as the second internal data source at the current moment; a fourth determination module, used to use the second data source at the current moment as the second target data source if the first internal data source at an adjacent moment of the current moment is the same as the first internal data source at the current moment; and a fifth determination module, used to use the first data source at the current moment as the second target data source if the second internal data source at an adjacent moment of the current moment is the same as the second internal data source at the current moment.

[0111] In a possible implementation, the above-mentioned device also includes: a historical moment data source acquisition module, which is used to obtain the first data source and the second data source of the historical moment; a sixth determination module, which is used to determine the correlation coefficient and the residual threshold through least squares fitting based on the first data source and the second data source of the historical moment; a target moment data source acquisition module, which is used to obtain the first data source and the second data source of the target moment in the historical moment; wherein each time period within the target moment includes a data source; the sixth determination module is used to determine the mutual correlation coefficient based on the data points of each time period.

[0112] In a possible implementation, the device further includes: a reporting module, configured to report a third target data source; wherein the third target data source is a data source that has not failed between the first data source at the current moment and the second data source at the current moment.

[0113] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0114] The device for determining the fault data source in this embodiment is presented in the form of a functional unit, where the functional unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0115] The embodiment of the present invention also provides a computer device having the above Figure 3 The device for determining the source of the faulty data is shown.

[0116] See also Figure 4 , Figure 4 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 4 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 10 is taken as an example.

[0117] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0118] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0119] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0120] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0121] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0122] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0123] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0124] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for determining a fault data source, characterized in that: The method comprises: Acquire the first data source at the current moment, the second data source at the current moment, and related content information; wherein the related content information includes: a residual threshold, a correlation coefficient, and a mutual correlation coefficient; Determine a first target data source corresponding to the first data source at the current moment according to the correlation coefficient, the mutual correlation coefficient, and the first data source at the current moment; wherein the first target data source is an expected data source of the second data source at the current moment; If the residual value between the target data source and the second data source is greater than the residual threshold, determine the target area information; wherein the target area information is used to indicate the target area applied by the first data source at the current moment and the second data source at the current moment; According to the target area information, a second target data source is determined from the first data source at the current moment and the second data source at the current moment; wherein the second target data source is a data source where a fault occurs between the first data source at the current moment and the second data source at the current moment.

2. The method for determining a fault data source according to claim 1, characterized in that: Determining a second target data source from the first data source at a current moment and the second data source at a current moment according to the target area information includes: Determine, according to the target area, a first vector product between a first data source at a current moment and a first data source at a moment before the current moment; Determine, according to the target area, a second vector dot product between the second data source at a current moment and the second data source at a moment before the current moment; Compare the dot product of the first vector and the dot product of the second vector to generate a comparison result; wherein, when the comparison result is that the dot product of the first vector is greater than the dot product of the second vector, use the second data source at the current moment as the second target data source; when the comparison result is that the dot product of the first vector is less than the dot product of the second vector, use the first data source at the current moment as the second target data source.

3. The method for determining a fault data source according to claim 2, characterized in that: If the residual value between the target data source and the second data source is greater than the residual threshold, determining the target area information includes: If the residual value between the target data source and the second data source is greater than the residual threshold, detecting whether a first internal data source corresponding to the first data source at the current moment and a second internal data source corresponding to the second data source at the current moment are the same; If the first internal data source and the second internal data source are the same, the target area information is determined.

4. The method for determining a fault data source according to claim 3, characterized in that: The method further comprises: If the first internal data source and the second internal data source are not the same, obtaining the first internal data source and the second internal data source at adjacent moments of the current moment; Detecting whether the first internal data source at an adjacent time point of the current time point is the same as the first internal data source at the current time point; Detecting whether the second internal data source at an adjacent time point of the current time point is the same as the second internal data source at the current time point; If the first internal data source at the adjacent time of the current time is the same as the first internal data source at the current time, the second data source at the current time is used as the second target data source; If the second internal data source at the adjacent time point of the current time point is the same as the second internal data source at the current time point, the first data source at the current time point is used as the second target data source.

5. The method for determining a fault data source according to claim 1, characterized in that: The process of determining relevant content information includes: Acquire a first data source and a second data source at a historical moment; Determine the correlation coefficient and the residual threshold value by least square fitting according to the first data source and the second data source at the historical moment; Obtain a first data source and a second data source of a target moment in the historical moment; wherein each time period in the target moment includes a data source; Based on the data points in each time period, the mutual correlation coefficient is determined.

6. The method for determining a fault data source according to any one of claims 1 to 5, characterized in that: The method further comprises: The third target data source is reported; wherein the third target data source is a data source that has not failed between the first data source at the current moment and the second data source at the current moment.

7. A device for determining a fault data source, characterized in that: The device comprises: An acquisition module, used to acquire the first data source at the current moment, the second data source at the current moment, and related content information; wherein the related content information includes: a residual threshold, a correlation coefficient, and a mutual correlation coefficient; A first determination module is used to determine a first target data source corresponding to the first data source at the current moment according to the correlation coefficient, the mutual correlation coefficient, and the first data source at the current moment; wherein the first target data source is an expected data source of the second data source at the current moment; A second determination module is used to determine target area information if the residual value between the target data source and the second data source is greater than the residual threshold; wherein the target area information is used to indicate the target area applied by the first data source at the current moment and the second data source at the current moment; The third determination module is used to determine a second target data source from the first data source at the current moment and the second data source at the current moment according to the target area information; wherein the second target data source is a data source where a fault occurs between the first data source at the current moment and the second data source at the current moment.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for determining the fault data source according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method for determining a fault data source according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to cause a computer to execute the method for determining a fault data source according to any one of claims 1 to 6.

Images