Data detection methods, devices, electronic equipment and storage media

Abstract

This disclosure provides a data detection method, apparatus, electronic device, and storage medium, relating to the field of artificial intelligence technology, and particularly to the field of big data processing technology. The specific implementation scheme is as follows: based on the service indicators of each data point in the dataset, determine the stability evaluation parameters of the dataset; based on the stability evaluation parameters of the dataset, determine a target quantile from a preset set of quantiles; based on the anomaly detection parameters of each data point in the dataset and the target quantile, identify anomalous data in the dataset. Using the embodiments of this disclosure, anomalous data can be accurately detected and the stability of the dataset can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and more particularly to the field of big data technology, specifically to a data detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the promotion and application of microservices and cloud-native technologies, distributed systems are becoming increasingly widespread. Generally speaking, the more types of services a distributed system provides, the more modules it includes. The intricate call chains and the complex module runtime environment of the cloud-native era pose significant challenges to the overall stability and response speed of the system. Among these challenges, data anomaly detection in distributed systems has a significant impact on system availability and speed, thus becoming a critical issue. Summary of the Invention

[0003] This disclosure provides a data detection method, apparatus, electronic device, and storage medium.

[0004] According to one aspect of this disclosure, a data detection method is provided, comprising:

[0005] Based on the service metrics of each data point in the dataset, determine the stability assessment parameters of the dataset;

[0006] Based on the stability evaluation parameters of the dataset, a target quantile is determined from a set of preset quantiles;

[0007] Based on the anomaly detection parameters of each data point in the dataset and the target quantile, abnormal data are identified in the dataset.

[0008] According to another aspect of this disclosure, a data detection apparatus is provided, comprising:

[0009] The stability assessment module is used to determine the stability assessment parameters of the dataset based on the service metrics of each data point in the dataset.

[0010] The quantile determination module is used to determine a target quantile from a set of preset quantiles based on the stability evaluation parameters of the dataset.

[0011] An anomaly detection module is used to identify anomalous data in the dataset based on anomaly detection parameters for each data point in the dataset and the target quantile.

[0012] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0013] At least one processor; and

[0014] The memory is communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods in any embodiment of this disclosure.

[0016] According to another aspect of this disclosure, a data processing system is provided, including an electronic device according to any embodiment of this disclosure.

[0017] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform the methods of any embodiment of this disclosure.

[0018] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods of any embodiment of this disclosure.

[0019] According to the technology disclosed herein, based on the service metrics of each data point in the dataset, the stability evaluation parameters of the dataset are obtained through overall evaluation. Based on the stability evaluation parameters, the target quantile can be determined. Then, based on the anomaly detection parameters of each individual data point in the dataset and the target quantile, abnormal data in the dataset can be accurately identified.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0021] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0022] Figure 1 This is a flowchart of the data detection method according to the first embodiment of this disclosure;

[0023] Figure 2 This is an architectural block diagram of the data processing system according to the second embodiment of this disclosure;

[0024] Figure 3 This is a flowchart of the data detection method according to the third embodiment of this disclosure;

[0025] Figure 4 This is a structural block diagram of the data processing apparatus according to the fourth embodiment of this disclosure;

[0026] Figure 5 This is a structural block diagram of the data processing apparatus according to the fifth embodiment of this disclosure;

[0027] Figure 6 This is a block diagram of an electronic device that implements the data detection method of the embodiments of this disclosure. Detailed Implementation

[0028] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0029] In distributed system applications, the more service types a system provides to users, the more complex its structure and the more modules it includes. In such cases, providing a single service might involve hundreds of modules within the system. The intricate call chains and runtime environment pose significant challenges to system stability and response speed. For high-availability systems, the persistent presence of some abnormal data can affect system availability, stability, and response speed. Therefore, detecting and self-healing abnormal data is crucial for preventing service performance degradation and maintaining system stability. Self-healing refers to deleting abnormal data and adding corresponding normal data.

[0030] The methods for anomaly detection in data can be mainly divided into two categories:

[0031] Firstly, the manual threshold detection method. This involves configuring different indicator collection items for the data used by different modules in the system. Based on experience, corresponding thresholds are set for different indicator items of the data. When a certain indicator of a certain data exceeds its corresponding threshold, the relevant staff are notified, and multiple indicators of the data are manually combined to determine whether the data is abnormal.

[0032] Secondly, the indicator model method. This method establishes different indicator models for different modules within the system, using these models to detect anomalies in the data used by each module. Common module models include the z-score method, the multivariate Gaussian method, and manually labeled supervised learning. The Gaussian method collects indicators from the data, builds a model using these indicators, and then determines whether each data point is abnormal by calculating the degree of deviation in the indicator distribution. Supervised learning methods require manually selecting training samples to train the corresponding indicator model. The indicators of the data to be detected are then input into the trained indicator model to determine whether the data is abnormal.

[0033] However, the above-mentioned anomaly detection methods all have certain limitations.

[0034] For manual threshold detection, this method requires configuring multiple monitoring indicators for each module in the system and setting corresponding thresholds based on experience. As the system's operations iterate, some indicators may become outdated, requiring periodic adjustments. The alarm thresholds for each indicator also need adjustment. If there are too many alarms, the manpower cost for manually checking for anomalies is relatively high. Moreover, the effectiveness of this method is highly dependent on the experience of the personnel involved, making it difficult to guarantee the effectiveness and real-time nature of anomaly detection.

[0035] While indicator-based methods can automatically detect anomalies in data, they still have several shortcomings. Supervised learning methods require manual labeling of training samples. The models lack versatility, as a single model cannot be used to detect anomalies across different modules within a system. Furthermore, labeling samples and adjusting model parameters are labor-intensive. As system operations evolve, the model's effectiveness gradually decreases. Moreover, with limited real-world data, individual anomalies can easily influence model parameters, potentially causing the model to miss anomalies in practical applications. In contrast, Gaussian methods determine the presence of anomalies by comparing indicators across different data points. While this method does not require manual sample labeling or model training, the indicators it detects must conform to a Gaussian distribution to effectively detect anomalies. In real-world applications, the distribution of each indicator in a system is diverse, and indicators conforming to a Gaussian distribution are relatively few.

[0036] Therefore, this disclosure provides a data detection scheme that can accurately detect whether there are anomalies in each data in the dataset, thereby improving the stability of the dataset.

[0037] Figure 1 A flowchart of a data detection method according to a first embodiment of this disclosure is shown. Figure 1 As shown, the method may include the following steps:

[0038] S110, Based on the service metrics of each data point in the dataset, determine the stability assessment parameters of the dataset;

[0039] S120, based on the stability evaluation parameters of the dataset, determines the target quantile among a set of preset quantiles;

[0040] S130, based on the anomaly detection parameters and target quantiles for each data point in the dataset, identifies anomalous data in the dataset.

[0041] For example, the data to be accessed by each module in a data processing system can be stored in the same dataset. Data from different modules may use different metrics to measure anomalies. Each data point may include multiple metrics, and each metric may be acquired in a different way. Data from different modules is captured, and information such as the data's identifier, source, and capture timestamp is recorded. Metrics or features are associated with the data and stored in the dataset. This allows for the retrieval and detection of data from different modules and their associated metric parameters using a unique identifier.

[0042] For example, a dataset may include one or more data sets. The dataset may record service metrics for each data set. When a data processing system calls data from the dataset, it collects the service metrics. Service metrics are characteristics or indicators that characterize the data's performance in providing services or responding to service requests. For example, service metrics may include the data's response time to a service request, whether the call was successful, service traffic, and the service node corresponding to the data.

[0043] In step S110 above, the overall service metrics corresponding to each data point in the dataset can be determined based on the service metrics of that data point. Examples include the average response time, long-tail distribution, call success rate, and average service traffic of the data in the dataset. Based on the overall service metrics, stability assessment parameters for the dataset can be determined. These stability assessment parameters characterize the stability with which the dataset provides services for the called services.

[0044] For example, multiple preset quantiles can be adjusted as needed. The preset multiple quantiles may include quartiles, octaves, hexadecimals, thirtieth-second quantiles, sixtieth-fourth quantiles, etc.

[0045] In some embodiments, the stability assessment parameter of the dataset and the target quantile value can be positively correlated, such as linear or exponential positive correlation. A higher value for the stability assessment parameter indicates greater stability of the dataset, and a higher target quantile value is used.

[0046] For example, the stability evaluation parameter of the dataset can range from 1 to 10. If the stability evaluation parameter is 5, the 16th percentile can be selected; if the stability evaluation parameter is 8, the 32nd percentile can be selected.

[0047] For example, the stability evaluation parameters for a dataset can be categorized into five levels, from level 1 to level 5. Higher levels indicate greater stability of the dataset. Correspondingly, the preset quantiles can include five different quantiles, such as the quartile, octave, 16th quartile, 32nd quartile, and 64th quartile. If the stability evaluation parameter is level 1, the target quantile is the quartile. If the stability evaluation parameter is level 2, the target quantile is the octave. If the stability evaluation parameter is level 3, the target quantile is the 16th quartile. If the stability evaluation parameter is level 4, the target quantile is the 32nd quartile. If the stability evaluation parameter is level 5, the target quantile is the 64th quartile.

[0048] In some embodiments, the target quantile can also be adjusted based on the size of the dataset. If the dataset size is less than a set threshold and the target quantile is higher than the set threshold, the target quantile is adjusted. For example, if the dataset size is small, such as 10 data points, and the target quantile is the 16th percentile, the 16th percentile cannot be used for anomaly detection because it exceeds the dataset size. In this case, the target quantile can be adjusted to the quartile or octet.

[0049] For example, anomaly detection parameters are parameters used to detect whether data is abnormal. Anomaly detection parameters may include data response time, service duration, etc. For each anomaly detection parameter, a target quantile can be used to identify abnormal data within the dataset for that specific anomaly detection parameter.

[0050] In this embodiment, the overall stability of the dataset can be evaluated using the service metrics of each data point, and this stability can then be used to determine the target quantile. When detecting anomalies in each data point, the target quantile can be compared with the anomaly detection parameters of each data point to determine whether each data point is anomaly. Since the value of the target quantile takes into account the overall stability of the dataset, the accuracy of anomaly detection can be improved when detecting whether each data point in the dataset is anomaly, and the stability of the dataset can be further improved.

[0051] In some embodiments, the average value of service metrics over different time periods can be used to assess the stability of the dataset.

[0052] For example, in step S110 above, determining the stability assessment parameters of the dataset based on the service metrics of each data point in the dataset may include:

[0053] Based on the service metrics for each data point in the dataset, determine the average value of the service metrics over at least one time period.

[0054] Based on the average value of service metrics over at least one time period, determine the stability assessment parameters for the dataset.

[0055] In this example, the average of the overall service metrics of the dataset over different time periods is used to assess the stability of the dataset and improve the accuracy of the assessment.

[0056] For example, taking service metric A as an example, the dataset includes data B1 to B10. The value of service metric A for each data point in data B1 to B10 is obtained at each time point, and the time period is divided into C1 to C3. Then, using the obtained values, the average value of service metric A within time period C1, the average value within time period C2, and the average value within time period C3 are calculated. Finally, the variation among these three average values ​​is used to determine the stability assessment parameters of the dataset.

[0057] When there are multiple service metrics, a mean change can be calculated by using the average change of each service metric over different time periods. Then, the stability evaluation parameters of the dataset can be determined using this mean change.

[0058] In some embodiments, the change in the average value of service metrics over two adjacent time periods can be used to determine the stability assessment parameters of the dataset.

[0059] For example, the above-mentioned determination of the stability assessment parameters of the dataset based on the average value of service metrics over at least one time period may include:

[0060] The stability assessment parameters of the dataset are determined based on the average value of the service metrics in a first time period and the average value in a second time period within at least one time period; wherein the first time period and the second time period are adjacent.

[0061] In this example, since changes within adjacent time periods better reflect the stability of the dataset, using the changes between the average values ​​of service metrics within two adjacent time periods to determine the stability assessment parameters of the dataset can improve the accuracy of the assessment.

[0062] In some embodiments, the change between the average value of the service metric in a first time period and the average value in a second time period is negatively correlated with the stability evaluation parameter of the dataset, such as linear negative correlation, exponential negative correlation, etc. The greater the change, the lower the value of the stability evaluation parameter, and the lower the stability of the dataset. Conversely, the smaller the change, the higher the value of the stability evaluation parameter, and the higher the stability of the dataset.

[0063] In some embodiments, the duration of the first time period and the second time period may be the same or different.

[0064] In some embodiments, data anomalies can be detected periodically. For example, detection can be performed every 12 hours or every 24 hours.

[0065] In some embodiments, the average value of the anomaly detection parameters for each data point can be collected periodically within that time period, and then the dataset can be periodically checked for anomalies.

[0066] For example, the anomaly detection parameters for each data point include the average value of the anomaly detection parameters for each data point within a third time period, wherein the third time period is adjacent to one of the first time period and the second time period, and the third time period is later than the first time period and the second time period.

[0067] In this example, the stability assessment parameters of the dataset can be determined periodically using service metrics from two adjacent historical time periods, thereby obtaining the target quantile. This quantile is then used to detect anomalies in each data point within the current time period, thus enabling dynamic adjustment of the quantile and dynamic detection of data anomalies.

[0068] For example, the first time period, the second time period, and the third time period have the same duration.

[0069] For example, when detecting data in the Nth time period, the changes in the average service metric in the (N-1)th time period and the average service metric in the (N-2)th time period can be compared to obtain the stability assessment parameters of the dataset. Then, based on the stability assessment parameters of the dataset, the target quantile for the Nth time period is determined. Based on the average anomaly detection parameters of each data point in the dataset in the Nth time period and the target quantile for the Nth time period, outlier data in the dataset is identified.

[0070] In this example, the time period can be 12 hours, 24 hours, or 36 hours, etc.

[0071] In some embodiments, the corresponding quantile can be determined in the anomaly detection parameters based on the target quantile, and then the abnormal data can be identified in the dataset based on the quantile and the anomaly detection parameters for each data point.

[0072] For example, step S130 above, which identifies anomalous data in the dataset based on the anomaly detection parameters and target quantiles for each data point, includes:

[0073] For each data point in the dataset, extract the anomaly detection parameters located at the target quantile.

[0074] Based on the anomaly detection parameters located at the target quantile, determine the quantile interval of the anomaly detection parameters;

[0075] Based on the anomaly detection parameters and quantiles of each data point in the dataset, abnormal data are identified in the dataset.

[0076] In this example, the quantile interval is further determined based on the target quantile. Using the quantile interval and the anomaly detection parameters for each data point, it can be determined whether each data point is an anomaly, effectively improving the accuracy of anomaly detection.

[0077] For example, taking the target quantile as the quartile as an example. Each data point has a corresponding anomaly detection parameter, and the anomaly detection parameter for each data point is collected. Anomaly detection parameters located at the quartile are extracted from all collected anomaly detection parameters. Based on the difference between the maximum and minimum values ​​of the anomaly detection parameters located at the quartile, the quartile distance of the anomaly detection parameters is determined. This difference is positively correlated with the quartile distance; the larger the difference, the larger the quartile distance. A comparison threshold is determined based on the quartile distance. This comparison threshold is compared with the anomaly detection parameter of each data point, and data whose anomaly detection parameter is greater than the comparison threshold are identified as anomaly data.

[0078] For example, the comparison threshold is equal to the quantile distance, or the comparison threshold is equal to three times the quantile distance plus the maximum value of the anomaly detection parameters located at the target quantile.

[0079] In some embodiments, for anomaly detection parameters that cannot be detected using quantiles, other detection methods can be employed. For example, to detect whether the calling port of the data is abnormal, the following methods can be used:

[0080] Retrieve the call status of each data point in the dataset within a specified time period;

[0081] Based on the call history of each data item within a set time period, determine whether there are any abnormal call ports for each data item.

[0082] For example, the call details may include the service being called and the data returned to the service being called.

[0083] For example, if, within a set time period, a first service request calls the first data but the first data is not successfully returned to the first service, then the first data can be determined to be abnormal data for the calling port. Alternatively, this port abnormality can be considered as a port outage. Conversely, if, within the set time period, for any service request, the first data is called and successfully returned to the corresponding service request, then the first data can be determined to be normal data for the calling port. Alternatively, this port can be considered as connected.

[0084] In this example, by detecting the data's call status within a set time period, it is possible to accurately detect whether the data has an abnormal call port.

[0085] Different anomaly detection parameters result in different anomaly types in the detected data. For each anomaly type, an update strategy is determined based on that specific anomaly type.

[0086] For example, exception types may include response time exceptions, call port exceptions, etc. Different exception data may correspond to the same or different exception types. For instance, the first exception data includes response time exceptions, the second exception data includes both response time exceptions and call port exceptions, and the third exception data includes only call port exceptions. The first, second, and third exception data are just one type of exception data, and they are not necessarily arranged in the order of first, second, and third. The update strategies used for exception data with different exception types also differ.

[0087] For example, data in the dataset can be checked and updated periodically. Alternatively, checks can be performed on requests, and abnormal data can be updated after the checks are completed.

[0088] In this example, determining the update strategy for each abnormal data point based on its corresponding abnormal type can improve the compatibility of dataset updates.

[0089] In some embodiments, the anomaly detection parameter includes response time.

[0090] For example, step S130 above may include: determining at least one data in the dataset that was called by the first calling service; obtaining at least one response time corresponding to each of the at least one data based on the first calling service; and determining data with abnormal response times in the at least one data based on the at least one response time and a target quantile.

[0091] In this example, the response time of data called for the same data can be used to detect whether there are any response time anomalies, thereby improving the accuracy of response time anomaly detection.

[0092] For example, the first calling service refers to one service within the calling services. Different calling services can call different data, and there can be overlap in their data calls. For instance, if the first calling service calls the first data and the second data in the dataset, and the second calling service calls the second data in the dataset, then the second data is the intersection of the data calls made by the first and second calling services. The response time for data to different calling services also varies. The response time for different data to the same calling service also varies. Therefore, in this embodiment of the disclosure, the detection of whether there is an anomaly in the response time of each data in the dataset can refer to its corresponding calling service.

[0093] For example, the response time mentioned above can be the average response time of the first call to the service, the median response time, the standard deviation of the response time, or the variance of the response time.

[0094] In practical applications, the data invoked by the first service call may include one or more items. If the data invoked by the first service call includes only one item, the system determines whether the data exhibits any response time anomalies based on the response time of that data and a set response time threshold.

[0095] If the data in the first service call includes multiple data items, a response time threshold is determined based on the response time of each data item in the first service call; based on the determined response time threshold, data items with abnormal response times are identified from the multiple data items in the first service call.

[0096] In some embodiments, an octet determination method can be used to determine a response time threshold based on the response time of each data item in the first service call. The determined response time threshold is the octet of the response times of all data items in the first service call.

[0097] For example, the target quantile is the quartile, and the above-mentioned data that identifies response time anomalies in at least one set of data based on at least one response time and the target quantile includes:

[0098] Extract the octet of the response time in at least one response time;

[0099] Based on the response time located at the octet, determine the octet of at least one response time;

[0100] Based on the interoctet and at least one response time, identify data with abnormal response times in at least one dataset.

[0101] In this example, the interoctet of the response time can be used to determine whether there are response time anomalies in the data, improving the accuracy of response time anomaly detection. Moreover, since this at least one response time is dynamic, the interoctet is also dynamic, which allows for dynamic adjustment of the time threshold used to determine whether there are response time anomalies in the data, further improving the accuracy of response time anomaly detection.

[0102] For example, the at least one response time is arranged to determine the octet. The octet response times comprise seven data points. The absolute value of the difference between the response time of the first octet and the response time of the seventh octet can be determined as the octet interval of these at least one response time. The response time of each of these at least one data points is compared to the octet interval to determine if any data point exhibits a response time anomaly. For example, data points with response times greater than the octet interval are identified as having an abnormal response time.

[0103] For example, the at least one response time is arranged, and the arrangement is divided into two parts by the median response time, where the first part has a longer response time than the second part. The first and second parts are then arranged from shortest to longest response time. The response time at the third quartile is extracted from the first part, and the response time at the first quartile is extracted from the second part. The difference between the third quartile and the first quartile can be determined as the octet of the at least one response time. The third quartile response time refers to the response time ranked third among the quartiles in the first part. The first quartile response time refers to the response time ranked first among the quartiles in the second part.

[0104] The following will describe, with examples, the data update strategy provided in the embodiments of this disclosure.

[0105] In some embodiments, for data with abnormal response time, it is first determined whether the amount of normal data in the dataset meets the set conditions. If the set conditions are not met, the corresponding normal data needs to be added first and then the abnormal data needs to be deleted. If the set conditions are met, the order between adding the corresponding normal data and deleting the abnormal data is not limited.

[0106] For example, the above-mentioned determination of the update strategy for each abnormal data based on the abnormal type corresponding to each of the at least one abnormal data may include:

[0107] If the anomaly type corresponding to the first abnormal data in at least one abnormal data includes response time anomaly but does not include call port anomaly, the update strategy corresponding to the first abnormal data is determined as the first strategy; wherein, the first strategy includes adding corresponding normal data to the dataset and deleting the first abnormal data based on the first abnormal data when the amount of normal data in the dataset does not meet the set conditions.

[0108] In this example, for data with abnormal response time but normal call port, if the amount of normal data in the dataset does not meet the set requirements, the corresponding normal data is added first and then the abnormal data is deleted. This can prevent the dataset from being unable to provide enough data for the system to call.

[0109] For example, the above-mentioned addition of corresponding normal data to the dataset based on the first abnormal data includes: extracting corresponding data from the source of the first abnormal data based on the calling service to which the first abnormal data applies, and then adding the extracted data to the dataset. In some embodiments, it is first verified whether the extracted data can be called by the calling service to which it applies. If it can, it is determined to be normal data and added to the dataset; if it cannot, it is determined to be abnormal data and cannot be added to the dataset.

[0110] In some embodiments, for data with an abnormal calling port, it is necessary to stop the calling service for that data first to avoid system failure due to the abnormal calling port of that data.

[0111] For example, the above-mentioned determination of the update strategy for each abnormal data based on the abnormal type corresponding to each of the at least one abnormal data may include:

[0112] If the exception type corresponding to the second exception data in at least one exception data includes a call port exception, the update strategy for the second exception data is determined to be the second strategy; wherein, the second strategy includes updating the second exception data based on the third strategy after stopping the call service of the second exception data; the third strategy includes adding corresponding normal data to the dataset based on the second exception data and deleting the second exception data if the amount of normal data in the dataset does not meet the set conditions.

[0113] In this example, for data with an abnormal calling port, the calling service for that data is stopped first to avoid call failures due to port abnormalities when a calling service is available to access that data. Furthermore, after stopping the calling service for that data, if the amount of normal data in the dataset does not meet the set requirements, the corresponding normal data is added first, and then the abnormal data is deleted. This prevents the dataset from being unable to provide enough data for the system to call.

[0114] For example, the above-mentioned addition of corresponding normal data to the dataset based on the second abnormal data includes: extracting corresponding data from the source of the second abnormal data based on the calling service to which the second abnormal data applies, and then adding the extracted data to the dataset. In some embodiments, it is verified whether the extracted data can be invoked by the calling service to which it applies. If it can, it is determined to be normal data and added to the dataset; if it cannot, it is determined to be abnormal data and discarded.

[0115] In some embodiments, when the data processing system is unable to delete abnormal data in the dataset or stop the call service for abnormal data due to its own reasons, the access domain name of the abnormal data is deleted or blocked in the domain name system corresponding to the dataset, thereby achieving the effect of indirectly deleting the abnormal data.

[0116] For example, the above update strategy may also include:

[0117] If it is impossible to delete the third abnormal data in at least one abnormal data set or to stop the service calling the third abnormal data set, then block or delete the access domain name of the third abnormal data set in the domain name system corresponding to the dataset.

[0118] In this example, the domain name for accessing abnormal data can be blocked or deleted in the domain name system, thus indirectly deleting the abnormal data. In this way, the system will not call the abnormal data because it cannot access it.

[0119] In some embodiments, the above update strategy may further include:

[0120] For abnormal data in the dataset, the domain names associated with the abnormal data are blocked from accessing the dataset within a set time period in the corresponding domain name system. This avoids blocking access to abnormal data for an extended period. If the blocking settings expire, the domain name system can restore normal access to the data.

[0121] In this embodiment of the disclosure, different update strategies are adopted for abnormal data of different abnormal types, which helps to improve the compatibility of dataset updates.

[0122] The embodiments disclosed herein can automate data acquisition, data detection, and abnormal data self-healing. The embodiments of this disclosure will be described below based on these three aspects.

[0123] Figure 2 An architectural block diagram of the data processing system provided in the second embodiment of this disclosure is shown.

[0124] The data processing system includes an acquisition system, an anomaly detection system, and a self-healing system.

[0125] The data acquisition system includes a timed trigger for data capture and data metrics. Different interfaces are used to collect data from different modules, resulting in a data list for each module. A unified data structure and data storage method are maintained. For example, metrics such as data throughput, response time, and port connectivity are collected and stored in a database, associating these metrics with the data. Data identifiers, module affiliation, and timestamps are also associated and stored in the database. This allows for the storage of data and its metrics with unique identifiers, facilitating data monitoring and updates, and improving data retrieval efficiency.

[0126] The anomaly detection system periodically extracts data and data metrics from the database to perform anomaly detection. It primarily employs the octet detection algorithm and the port connectivity detection algorithm to detect anomalies.

[0127] The self-healing system employs different update strategies to update the database for different types of abnormal data. For example, it may first expand the abnormal instance and then delete it, suspend the instance's service calls and expand the abnormal instance before deleting it, or delete or block the access domain name of the abnormal instance in the domain name system.

[0128] Figure 3 A flowchart of a data detection method according to a third embodiment of this disclosure is shown.

[0129] The anomaly detection system periodically reads data and its metrics from the database and performs data analysis using both the quartile detection algorithm and the port connectivity detection algorithm to identify anomalous data. The quartile detection algorithm is used to detect data response time, resulting in a list of data with excessively slow response times, such as a list of slow instances. The port connectivity detection algorithm is used to analyze data call patterns, resulting in a list of data with port anomalies, such as a list of instances with port connectivity issues.

[0130] In practical applications, taking the education business scenario as an example, when a user accesses education services in the system, the system responds to the request by calling an instance in backend A. For example, backend A has 20 instances, and the upstream system's response time setting for the backend is: if the response time of any instance exceeds 300ms, the response time of that instance is determined to have timed out. If the response time of instances in the backend frequently times out, the upstream system cannot retrieve instances from the backend, thus causing the user's request to fail. Even if the upstream system has a retry policy for backend A, if an abnormal instance in backend A is accessed, the retry request will fail directly, which will increase the number of upstream retries and increase the probability of access failure. The method provided in this embodiment can periodically detect and update abnormal instances, thereby ensuring the stability of the education service.

[0131] Figure 4 A structural block diagram of a data detection apparatus according to a fourth embodiment of this disclosure is shown. Figure 4 As shown, the data detection device may include:

[0132] Stability assessment module 410 is used to determine the stability assessment parameters of the dataset based on the service metrics of each data point in the dataset.

[0133] Quantile determination module 420 is used to determine a target quantile from a set of preset quantiles based on the stability evaluation parameters of the dataset.

[0134] Anomaly detection module 430 is used to identify anomalous data in the dataset based on anomaly detection parameters for each data point in the dataset and the target quantile.

[0135] Figure 5 A structural block diagram of a data detection apparatus according to a fifth embodiment of this disclosure is shown. Figure 5 As shown, the data detection device includes a stability assessment module 510, a quantile determination module 520, and an anomaly detection module 530, which have the same functions as the stability assessment module 410, quantile determination module 420, and anomaly detection module 430 in the above embodiments, and will not be described in detail here.

[0136] For example, the stability evaluation module 510 includes:

[0137] The mean calculation unit 511 is used to determine the average value of the service indicator over at least one time period based on the service indicator of each data in the dataset.

[0138] The evaluation parameter determination unit 512 is used to determine the stability evaluation parameters of the dataset based on the average value of the service indicator over the at least one time period.

[0139] For example, the evaluation parameter determination unit 512 is used to:

[0140] Based on the average value of the service metric in a first time period and the average value in a second time period within the at least one time period, the stability evaluation parameters of the dataset are determined; wherein the first time period and the second time period are adjacent.

[0141] For example, the anomaly detection parameters for each data point include the average value of the anomaly detection parameters for each data point within a third time period, wherein the third time period is adjacent to one of the first time period and the second time period, and the third time period is later than the first time period and the second time period.

[0142] For example, the first time period, the second time period, and the third time period have the same duration.

[0143] For example, the anomaly detection module 530 includes:

[0144] The indicator extraction unit 531 is used to extract the anomaly detection parameters located at the target quantile from the anomaly detection parameters of each data in the dataset.

[0145] The quantile distance determination unit 532 is used to determine the quantile distance of the anomaly detection parameters based on the anomaly detection parameters located at the target quantile.

[0146] The abnormal data determination unit 533 is used to determine abnormal data in the dataset based on the abnormal detection parameters of each data in the dataset and the quantile.

[0147] The functions of each unit, module, or sub-module in the various devices of this disclosure embodiment can be found in the corresponding descriptions in the above method embodiments, and will not be repeated here.

[0148] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0149] According to embodiments of this disclosure, this disclosure also provides a data processing system that may include the electronic devices implemented in this disclosure.

[0150] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0151] like Figure 6As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. The RAM 603 may also store various programs and data required for the operation of the electronic device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0152] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of displays, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0153] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as data detection methods. For example, in some embodiments, the data detection method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via ROM 102 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the data detection method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the data detection method by any other suitable means (e.g., by means of firmware).

[0154] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0155] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable lane-changing control device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0156] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0157] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0158] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0159] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0160] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0161] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data detection method, comprising: Based on the service metrics of each data point in the dataset, stability evaluation parameters for the dataset are determined; wherein, the dataset is used to store the data to be called by various modules in the data processing system; Based on the stability evaluation parameters of the dataset, a target quantile is determined from a set of preset quantiles; Based on the anomaly detection parameters of each data point in the dataset and the target quantile, abnormal data are identified in the dataset. The determination of stability assessment parameters for the dataset based on service metrics for each data point in the dataset includes: Based on the service metrics for each data point in the dataset, determine the average value of the service metrics over at least one time period; Based on the average value of the service metric in a first time period and the average value in a second time period within the at least one time period, the stability evaluation parameters of the dataset are determined; wherein the first time period and the second time period are adjacent.

2. The method according to claim 1, wherein, The anomaly detection parameters for each data point include the average value of the anomaly detection parameters for each data point within a third time period, wherein the third time period is adjacent to one of the first time period and the second time period, and the third time period is later than the first time period and the second time period.

3. The method according to claim 2, wherein, The first time period, the second time period, and the third time period have the same duration.

4. The method according to any one of claims 1 to 3, wherein, The process of identifying anomalous data in the dataset based on the anomaly detection parameters for each data point in the dataset and the target quantile includes: In the anomaly detection parameters of each data point in the dataset, extract the anomaly detection parameters located at the target quantile; Based on the anomaly detection parameters located at the target quantile, the quantile distance of the anomaly detection parameters is determined; Based on the anomaly detection parameters of each data point in the dataset and the quantile, abnormal data are identified in the dataset.

5. A data detection device, comprising: The stability assessment module is used to determine the stability assessment parameters of the dataset based on the service metrics of each data point in the dataset; wherein, the dataset is used to store the data to be called by various modules in the data processing system; The quantile determination module is used to determine a target quantile from a set of preset quantiles based on the stability evaluation parameters of the dataset. An anomaly detection module is used to identify anomalous data in the dataset based on anomaly detection parameters for each data point in the dataset and the target quantile. The stability evaluation module includes: The mean calculation unit is used to determine the average value of the service indicator over at least one time period based on the service indicator of each data point in the dataset. The evaluation parameter determination unit is used to determine the stability evaluation parameters of the dataset based on the average value of the service indicator in a first time period and the average value in a second time period within the at least one time period; wherein the first time period and the second time period are adjacent.

6. The apparatus according to claim 5, wherein, The anomaly detection parameters for each data point include the average value of the anomaly detection parameters for each data point within a third time period, wherein the third time period is adjacent to one of the first time period and the second time period, and the third time period is later than the first time period and the second time period.

7. The apparatus according to claim 6, wherein, The first time period, the second time period, and the third time period have the same duration.

8. The apparatus according to any one of claims 5 to 7, wherein, The anomaly detection module includes: The indicator extraction unit is used to extract the anomaly detection parameters located at the target quantile from the anomaly detection parameters of each data in the dataset. The quantile distance determination unit is used to determine the quantile distance of the anomaly detection parameters based on the anomaly detection parameters located at the target quantile. An abnormal data determination unit is used to determine abnormal data in the dataset based on the abnormal detection parameters of each data in the dataset and the quantile.

9. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-4.

10. A data processing system comprising the electronic device as described in claim 9.

11. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method of any one of claims 1-4.

12. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-4.

Images