Burr data processing method, device, equipment, medium and program product

Abstract

This application provides a method, apparatus, device, storage medium, and program product for processing glitch data, which can be applied in the field of big data technology. The method includes: acquiring data sequence values ​​collected according to a time scale within a test time period; acquiring baseline data sequence values ​​collected according to a time scale within a comparison time period; in response to the positive deviation between the current data sequence value and the baseline data sequence value at the comparison time exceeding a first threshold, marking the current time as a candidate abnormal time, wherein the current time is selected from the test time period, and the comparison time is the time corresponding to the current time within the comparison time period; calculating the data change rate at the current time; in response to the data change rate exceeding a second threshold, determining the candidate abnormal time as a glitch time; and based on the baseline data sequence value at the comparison time, splitting the data sequence value of the glitch time into a data component and a glitch data component, and removing the glitch data component.

Description

Technical Field

[0001] This application relates to the field of big data technology, and to a method, apparatus, equipment, medium, and program product for processing glitch data. Background Technology

[0002] During routine data collection and analysis, outliers may appear at certain times. These outliers distort the statistical distribution of data, appearing as "spikes" or "noise" on charts. Such outliers reduce data quality, interfere with the analysis process, diminish the value of the analysis, and can even mislead data-driven decision-making. Summary of the Invention

[0003] In view of the above problems, embodiments of this application provide a method, apparatus, device, medium, and program product for processing burr data.

[0004] According to a first aspect of this application, a method for processing glitch data is provided, comprising: acquiring data sequence values ​​collected according to a time scale within a test time period; acquiring a reference data sequence value collected according to the time scale within a comparison time period; in response to a positive deviation between the data sequence value at the current moment and the reference data sequence value at the comparison time exceeding a first threshold, marking the current moment as a candidate abnormal moment, wherein the current moment is selected from the test time period, and the comparison moment is a moment in the comparison time period corresponding to the current moment; calculating the data change rate at the current moment, and in response to the data change rate exceeding a second threshold, determining the candidate abnormal moment as a glitch moment; and splitting the data sequence value of the glitch moment into a data component and a glitch data component based on the reference data sequence value at the comparison moment, and removing the glitch data component.

[0005] According to an embodiment of this application, the comparison time period is a historical time period without glitch, and the comparison time period includes a single historical time period that is an integer number of statistical periods apart from the time period to be tested, or the average of N single historical time periods, where N is an integer greater than 1.

[0006] According to an embodiment of this application, calculating the data change rate at the current moment includes: calculating the change rate of the data sequence value at the current moment relative to the data sequence value at an adjacent moment, wherein the adjacent moment is a moment at a predetermined time interval from the current moment.

[0007] According to an embodiment of this application, the adjacent time includes an earlier time and a later time, the earlier time is earlier than the current time, and the later time is later than the current time. The data change rate exceeding the second threshold includes: the rate of increase of the data sequence value at the current time relative to the data sequence value at the earlier time exceeds the second threshold, and the rate of decrease of the data sequence value at the current time relative to the data sequence value at the later time exceeds the second threshold.

[0008] According to an embodiment of this application, the first threshold is a first fixed value, or is calculated based on the benchmark data sequence value of the comparison time period; the second threshold is a second fixed value, or is calculated based on the benchmark data sequence value of the comparison time period.

[0009] According to an embodiment of this application, the step of splitting the data sequence value at the glitch moment into a data component and a glitch data component based on the reference data sequence value at the comparison moment, and removing the glitch data component includes: accumulating the glitch data component at each glitch moment within the test time period to obtain the total number of glitches; obtaining the total data volume based on the data sequence value collected within the test time period; and removing the total number of glitches from the total data volume.

[0010] According to an embodiment of this application, the method further includes: diagnosing the data acquisition source based on the determined glitch time, marking the data record or reporting the abnormal process.

[0011] According to an embodiment of this application, the method further includes: extracting the data curve graph within the time period to be tested, comparing it with a pre-stored historical glitch graph template; and determining the time period to be tested as having glitch anomalies in response to the similarity comparison result exceeding a preset threshold.

[0012] According to a second aspect of this application, a glitch data processing apparatus is provided, comprising: a test data acquisition module for acquiring data sequence values ​​collected according to a time scale within a test time period; a reference data acquisition module for acquiring reference data sequence values ​​collected according to the time scale within a comparison time period; a candidate anomaly determination module for marking the current time as a candidate anomaly time in response to a positive deviation between the data sequence value at the current time and the reference data sequence value at the comparison time exceeding a first threshold, wherein the current time is selected from the test time period, and the comparison time is a time in the comparison time period corresponding to the current time; a change rate calculation module for calculating the data change rate at the current time, and determining the candidate anomaly time as a glitch time in response to the data change rate exceeding a second threshold; and a glitch data removal module for splitting the data sequence value at the glitch time into a data component and a glitch data component based on the reference data sequence value at the comparison time, and removing the glitch data component.

[0013] According to a third aspect of this application, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0014] According to a fourth aspect of this application, a computer-readable storage medium is also provided, on which a computer program or instructions are stored, wherein the computer program or instructions, when executed by a processor, implement the steps of the above-described method.

[0015] According to a fifth aspect of this application, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method. Attached Figure Description

[0016] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0017] Figure 1 The illustration shows an application scenario diagram of the burr data processing method, apparatus, device, medium, and program product according to embodiments of this application;

[0018] Figure 2 A flowchart illustrating a burr data processing method according to an embodiment of this application is shown schematically.

[0019] Figure 3 A schematic diagram of burr data according to an embodiment of this application is shown;

[0020] Figure 4This schematic diagram illustrates a structural block diagram of a burr data processing apparatus according to an embodiment of the present application;

[0021] Figure 5 A block diagram schematically illustrates an electronic device suitable for implementing a glitch data processing method according to an embodiment of this application. Detailed Implementation

[0022] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0023] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0024] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0025] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0026] In the process of routine data statistics, the presence of abnormal data can lead to distortion of the statistical distribution of data, affecting data quality and interfering with the data analysis process.

[0027] An embodiment of this application provides a method for processing glitch data, comprising: acquiring data sequence values ​​collected according to a time scale within a test time period; acquiring a baseline data sequence value collected according to the time scale within a comparison time period; in response to a positive deviation between the data sequence value at the current moment and the baseline data sequence value at the comparison time exceeding a first threshold, marking the current moment as a candidate abnormal moment, wherein the current moment is selected from the test time period, and the comparison moment is a moment in the comparison time period corresponding to the current moment; calculating the data change rate at the current moment, and in response to the data change rate exceeding a second threshold, determining the candidate abnormal moment as a glitch moment; and splitting the data sequence value of the glitch moment into a data component and a glitch data component based on the baseline data sequence value at the comparison moment, and removing the glitch data component.

[0028] The abnormal data identification, removal, and restoration method based on burr features provided in this application can detect data quality, identify abnormal data, eliminate the influence of abnormal "burr" data, and ensure the reliability of the data used in the analysis.

[0029] The data processed in this application embodiment can possess characteristics of magnitude stability, temporal distribution stability, and statistical caliber stability. Magnitude stability indicates that the statistical data volume is sufficiently large and possesses a stable data scale, thus forming a stable statistical object. Temporal distribution stability indicates that the distribution of statistical data over time is stable; when data is collected from different dates, the data distribution at each moment is basically consistent, and the cumulative data curves over the time scale are similar to each other. Statistical caliber stability indicates that within the compared time span, the statistical caliber remains unchanged, or its changes have negligible impact.

[0030] Scale stability can be used to describe the stability of a business. Large-scale and steadily growing businesses exhibit scale stability; such businesses implicitly possess a stable trend, and statistically obtained data will fluctuate around this stable trend, with similarities outweighing differences. For rapidly growing businesses, it is recommended to use recent data as a reference. In practical applications, large-scale businesses with broad coverage and a large number of employees generally meet this condition, i.e., stability with a large sample size. However, businesses with a smaller audience or newly launched businesses do not meet this prerequisite.

[0031] Temporal distribution stability describes the stability of business operations over time, including stability across weekdays / holidays and the stability of the time scale. Stability across weekdays / holidays refers to the relatively stable comparability of business data over similar weekdays or holidays. Stability of the time scale refers to the relatively stable distribution structure of business data over a 24-hour period. In practical applications, large-scale businesses with broad coverage and a large number of employees typically meet this condition, which can be verified through statistical comparison.

[0032] Regarding the stability of statistical definitions, this prerequisite is quite intuitive: as long as the statistical object remains unchanged or its changes have minimal impact, the condition is met. Data with significant changes in statistical definitions are not comparable.

[0033] It should be noted that the glitch data processing method and apparatus provided in this application can be used in the field of big data technology, or in any field other than big data technology, such as the field of financial technology and the field of data statistics and anomaly detection technology. Here, the application field of the glitch data processing method and apparatus provided in this application is not limited.

[0034] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0035] In scenarios where personal information is used for automated decision-making, the methods, devices, and systems provided in this application all provide users with corresponding operation entry points for users to choose to agree to or reject the automated decision results; if the user chooses to reject, the process enters the expert decision-making process.

[0036] As used in this paper, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

[0037] It's important to note that the term "neural network" can refer to a machine learning network based on deep learning. A neural network processes input and provides corresponding output, typically consisting of an input layer, an output layer, and one or more hidden layers between them. Neural networks used in deep learning applications often include many hidden layers, increasing the network's depth. The layers of a neural network are connected sequentially, so that the output of the previous layer serves as the input to the next layer. The input layer receives the input to the neural network, while the output layer's output becomes the final output. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each processing the input from the layer above.

[0038] It should be understood that machine learning generally includes three phases: training, testing, and application (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively updating parameter values ​​until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as the input-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. In the application phase, the model can be used to process actual inputs based on the trained parameter values ​​to determine the corresponding output.

[0039] Figure 1 The diagram illustrates an application scenario of the burr data processing method according to an embodiment of this application. Figure 1 As shown, application scenario 100 according to an embodiment of this application may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables. For example, a user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send information, etc.

[0040] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be electronic devices such as smartphones, wearable devices, personal computers, intelligent voice interaction devices, smart home appliances, intelligent vehicles, in-vehicle terminals, aircraft, unmanned vending terminals, and extended reality devices. Extended reality devices can include virtual reality devices, augmented reality devices, and mixed reality devices. A client application for the target application can be installed and run on the terminal devices. This target application can include, but is not limited to, financial transaction applications, payment applications, shopping applications, web browser applications, search applications, instant messaging tools, email clients, and social media platform software (these are just examples). Furthermore, this application embodiment does not limit the form of the target application, and it can include, but is not limited to, applications, mini-programs, etc., installed on the terminal devices, and can also be in the form of web pages.

[0041] Server 105 can be a server providing various services, such as a backend management server supporting websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process received user requests and other data, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services such as cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and basic cloud computing services such as big data. The server can be the backend server of the aforementioned target application, used to provide backend services to the clients of the target application.

[0042] It should be noted that the glitch data processing method provided in this application embodiment can generally be executed by server 105 and / or terminal devices 101-103. Accordingly, the glitch data processing device provided in this application embodiment can generally be disposed in server 105 and / or terminal devices 101-103.

[0043] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0044] Figure 2 A flowchart illustrating a burr data processing method according to an embodiment of this application is shown schematically. Figure 2 As shown, the burr data processing method 200 according to an embodiment of this application may include steps S210 to S250.

[0045] It should be noted that, although the steps of the method are described in a specific order in the embodiments of this application, it should be understood that the execution order of these steps is not fixed and can be adjusted according to actual needs. Specifically, some steps can be executed in reverse order, or they can be executed in parallel to improve efficiency without affecting the overall method effect. In addition, some steps may be optional and can be selectively executed according to specific application scenarios or requirements.

[0046] In step S210, the data sequence values ​​collected according to the time scale within the time period to be tested are obtained.

[0047] In embodiments of this application, user consent or authorization can be obtained before acquiring user information. For example, a request to acquire user information can be sent to the user before step S210. If the user consents or authorizes the acquisition of user information, step S210 is executed.

[0048] In step S220, the baseline data sequence values ​​collected according to the time scale within the comparison time period are obtained.

[0049] In step S230, in response to the positive deviation between the data sequence value at the current time and the baseline data sequence value at the comparison time exceeding a first threshold, the current time is marked as a candidate abnormal time, wherein the current time is selected from the time period to be tested, and the comparison time is the time in the comparison time period that corresponds to the current time.

[0050] In step S240, the data change rate at the current moment is calculated, and in response to the data change rate exceeding the second threshold, the candidate abnormal moment is determined as a spike moment.

[0051] In step S250, based on the baseline data sequence value at the comparison time, the data sequence value at the glitch time is split into a data component and a glitch data component, and the glitch data component is removed.

[0052] In the embodiments of this application, a corresponding operation entry can be provided to the user, allowing the user to choose to agree to or reject the automated decision result. That is, before processing / making a decision to remove glitch data components, the user can provide an instruction to agree to or reject the processing / decision through the corresponding operation entry. If the user agrees to the processing / decision, the processing / decision to remove glitch data components is performed, i.e., step S250 is executed. If the user rejects the processing / decision, the expert decision-making process is initiated.

[0053] According to the embodiments of this application, by identifying spikes through dual judgment conditions and removing spike components based on comparison benchmarks, it is possible to automatically and accurately identify abnormal points that briefly surge and quickly fall back from time series data sequences, avoiding misjudging normal fluctuations or continuous step jumps as spikes; restoring a more realistic normal data sequence, and improving the reliability of subsequent statistical analysis and decision-making.

[0054] For example, the time period to be tested can represent the target time interval for which glitch detection, anomaly identification, and data restoration processing are required; for example, it could be 3 hours, 1 day, etc. (See reference...) Figure 3 Date 'a' (one day) is an example of the time period to be tested. Within this time period, the system collects business data at fixed time scales (e.g., every minute, every 5 minutes), forming data sequence values ​​(e.g., transaction volume, access volume, etc. at each moment). By comparing the data sequence values ​​within the time period to be tested with the baseline data sequence values ​​of the "comparison time period" (which will be explained below, but will not be elaborated on here), it is determined whether there are any spikes and rapid drops in data within the time period to be tested. These abnormal data are then marked, removed, or restored at least one of the following methods, ultimately outputting a clean data sequence.

[0055] A time scale represents the basic unit of time or time sampling point used when discretizing a continuous time axis. The system does not record data continuously without intervals, but rather collects data at fixed time intervals. This interval (e.g., 1 minute, 5 minutes, etc.) is the size of the time scale. For example, with a 1-minute time scale, the sampling times are 08:00, 08:01, 08:02, etc. In a time-sorted data sequence, each pair of specific sampling times constitutes a "time scale".

[0056] The identification of spikes requires comparison at the corresponding acquisition times. For example, comparing 08:00 on the test date 'a' with 08:00 on the comparison date 'b'. The rapid rise / fall of spikes also needs to be judged based on the rate of change of the data at the test time relative to the nearest acquisition time (referred to as "nearest time"). For example, the rate of change of the data at 20:03 relative to 20:00 is used to determine whether the data value at 20:03 is rising rapidly, or it can also be used to determine whether the data value at 20:00 is falling rapidly.

[0057] A data sequence value can represent the specific business data value collected at any given time. Data sequence values ​​can be statistical quantities generated by the business system at a specific moment, such as the number of transactions (e.g., 150 transactions were generated in the minute 10:00 (from 09:59 to 10:00)), page views (e.g., 1200 clicks in the minute 10:00), the number of data packets reported by the device, etc. Arranging the data values ​​collected according to the time scale in chronological order constitutes a data sequence. Figure 3 The schematic diagram of the spike data shown illustrates the data sequence for the test period (1 day). For clarity, Figure 3 The horizontal axis is marked with a time scale of 43 minutes, displaying the data sequence values ​​corresponding to each acquisition time.

[0058] According to an embodiment of this application, the comparison time period is a historical time period without spikes. The comparison time period includes a single historical time period that is an integer number of statistical periods apart from the time period to be tested, or the average of N single historical time periods, where N is an integer greater than 1.

[0059] For example, the comparison time period can represent a historical time period that is considered normal (without spikes or anomalies) and used for comparison with the "test time period". It provides a "benchmark" or "reference scale" for measuring whether the data is normal.

[0060] An integer number of statistical periods can represent an integer multiple of the statistical periods between the comparison period and the test period. For example, if the comparison period is the third statistical period before the test period, then the comparison period and the test period are separated by two statistical periods. The statistical period can be 1 day, 1 week, etc.

[0061] Reference Figure 3 We use "Date a" as the test period and "Date b" as the comparison period. Date b is the data obtained under normal circumstances (without abnormal spikes) and is comparable to date a (e.g., both are weekdays or specific periods of holidays).

[0062] Within the comparison period, data is collected using the same time scale as the period to be tested, forming a set of "baseline data sequence values". For example, if the data sequence values ​​for the period to be tested, from 20:00 to 22:00 on May 5th, are abnormal, the baseline data sequence values ​​for the period from 20:00 to 22:00 on April 5th of the same year, sampled every minute, can be compared with those of the baseline data sequence values.

[0063] The comparison time period should be normal, free of spikes, and comparable to the time period to be tested in terms of business statistical characteristics. The comparison time period can: provide a deviation benchmark for calculating whether the positive deviation between the current data sequence value and the normal benchmark data sequence value exceeds a first threshold; provide a decomposition basis, subtracting the benchmark data sequence value from the current data sequence value to obtain the spike data component when removing spikes; and ensure identification accuracy, as a comparison time period itself containing spikes can lead to misjudgments, therefore a time period without anomalies must be selected.

[0064] For example, when constructing a comparison period, the normality and comparability of the selected period should be emphasized. The comparison period can be the combined daily average of the same day last week, the same day last month, other working days (or holidays) this week, or other similar holiday days. The selection of the comparison period is relatively flexible; its normal characteristics should be highlighted.

[0065] In some embodiments, the comparison time period can be the average of multiple individual historical time periods. For example, if Friday, April 10th is the time period to be tested, the average of the data sequence values ​​at each collection time on Friday, March 27th and Friday, April 3rd of the same year, or the average of the data sequence values ​​at each collection time from April 6th to April 9th ​​of the same year, can be used as the baseline data sequence value at each collection time of the comparison time period.

[0066] The baseline data sequence value can be represented as the value of normal business data collected according to the time scale within a comparison period. It serves as a reference standard for measuring whether the data within the test period is abnormal. For example, if "the same day last week" is selected as the comparison period, then the business data at each time of that day (e.g., 20:00, 20:01, 20:02, etc.) are the baseline data sequence values.

[0067] The data sequence values ​​are the data sequence values ​​collected within the test time period (which may contain spikes). The baseline data sequence values ​​are the data sequence values ​​collected within the comparison time period (which are considered normal). The two correspond one-to-one on the same time scale; for example, 20:00 in the test time period corresponds to 20:00 in the comparison time period.

[0068] The benchmark data sequence value can be used to: calculate the positive deviation by subtracting the benchmark data sequence value of the corresponding comparison time from the data sequence value of the time to be measured, and determine whether the difference exceeds the first threshold; quantify the glitch component by subtracting the benchmark data sequence value of the corresponding comparison time from the data sequence value of that time, and the difference can be the glitch data component; and ensure the objectivity of the judgment benchmark by using the benchmark data sequence value from historical periods without anomalies, so that "normal data" has a quantifiable reference and avoids subjective settings.

[0069] According to embodiments of this application, ensuring the normality and comparability of the comparison benchmark improves the accuracy of identifying candidate abnormal moments and avoids misjudgments caused by spikes in the comparison benchmark itself. Constructing a benchmark data sequence value for the comparison period based on the average of multiple time periods helps smooth out random fluctuations and improves the stability and representativeness of the benchmark data sequence value.

[0070] For example, spike data is characterized by data spikes that are significantly higher than during normal periods. When spikes occur, the data sequence value (e.g., business volume) is significantly greater than the baseline data sequence value for the comparison period, which satisfies the following inequality.

[0071]

[0072] Where ywl(c) is the data sequence value at time c within the test time period, ywl(d) is the baseline data sequence value at time d within the comparison time period, and d is the time in the comparison time period corresponding to time c in the test time period. For definable coefficients, For reference and comparison standards.

[0073] It can be determined based on business sensitivity, data volatility, experience, or statistical methods. For example, it can be used for businesses with low tolerance for anomalies (e.g., transaction risk control). A smaller value can be set (e.g., 1.5 to 3) to reduce the false negative rate; for example, when business data fluctuates significantly. The value should be increased accordingly to avoid misjudging normal fluctuations as spikes; for example, it can be determined by analyzing the multiple of the maximum normal fluctuation amplitude in historical normal data. Alternatively, a statistical indicator (e.g., 3 standard deviations) can be used as a reference. Or, a large amount of training data can be used to train a neural network model to obtain [the necessary parameters]. .

[0074] This can be determined using baseline data sequence values ​​from the same point in a normal time period (corresponding to the current time) or the average of baseline data sequence values ​​from multiple normal time periods, emphasizing the business scale under normal conditions. Alternatively, a large amount of training data can be used to train a neural network model to obtain [the necessary data]. .

[0075] and The product of these two values ​​is the threshold for identifying candidate abnormal moments (spurt height). In other words, it can be used as the first threshold. For example, if the difference between the current traffic volume and the traffic volume at the comparison moment is higher than this threshold, it can be determined that the data at the current moment may be a spurt, and the current moment can be identified as a candidate abnormal moment.

[0076] According to an embodiment of this application, the first threshold is a first fixed value, or is calculated based on the benchmark data sequence value of the comparison time period; the second threshold is a second fixed value, or is calculated based on the benchmark data sequence value of the comparison time period.

[0077] The formula above shows the calculation of the first threshold based on the baseline data sequence value of the comparison period. This approach is applicable to scenarios where business volume fluctuates periodically, for example, where the first threshold is... and The product of the two. For scenarios with stable business volume and a clear normal fluctuation range, the first threshold can also be a first fixed value set by the staff. For example, a fixed value can be defined based on business sensitivity, data fluctuation range, experience or statistical methods as a standard to judge the degree of positive deviation between the current data sequence value and the baseline data sequence value at the comparison time, thereby simplifying the calculation process and speeding up the identification of candidate abnormal moments.

[0078] For example, the second threshold is used to determine whether the rate of change of the data sequence value within the test time period exceeds the normal range. It is the second condition for identifying spikes. The second threshold is for the rate of change of data rather than the absolute deviation.

[0079] The second threshold can be a fixed value or calculated based on the baseline data sequence values ​​of the comparison period. For scenarios with relatively stable rates of change, the second threshold can be a fixed value. For example, it can be set as a fixed value based on factors such as sensitivity to sudden events, the basic fluctuation of the business change rate, and coordination with a predetermined time (the interval between the current time and the nearest time). The calculation of the second threshold based on the baseline data sequence values ​​of the comparison period will be explained below, and will not be elaborated on further here.

[0080] According to embodiments of this application, fixed thresholds are suitable for scenarios with relatively stable traffic volume and rate of change, are simple to implement, and can improve response speed while saving system resources; dynamic thresholds can adapt to normal fluctuations in traffic volume and rate of change, reducing false alarms and missed alarms. The combined setting of the first threshold and the second threshold enables accurate characterization and identification of the specific abnormal pattern of spikes: the first threshold captures the amplitude of "surges," and the second threshold captures "intermittent" waveforms. The two work together to improve the accuracy, robustness, and business adaptability of abnormal data identification.

[0081] The first criterion for identifying glitch data using a dual-judgment system has been explained above: The current time is selected from the test time period, and a corresponding time from the comparison time period is selected as the comparison time. If the positive deviation between the data sequence value at the current time and the baseline data sequence value at the comparison time exceeds a first threshold, the current time is marked as a candidate abnormal time. To maintain detection accuracy, further judgment is needed on the candidate abnormal times to determine the glitch time. The candidate abnormal time is the time at which glitch data is highly likely to exist.

[0082] According to embodiments of this application, in order to further ensure the accuracy of glitch detection and prevent normal data sequence value changes from being identified as glitch, it is also necessary to determine the rate of change of the current data sequence value relative to the data sequence value at an adjacent time.

[0083] According to an embodiment of this application, calculating the data change rate at the current moment includes: calculating the change rate of the data sequence value at the current moment relative to the data sequence value at an adjacent moment, where the adjacent moment is a moment that is a predetermined time interval from the current moment.

[0084] According to an embodiment of this application, the adjacent time includes an earlier time and a later time, the earlier time is earlier than the current time, and the later time is later than the current time. The data change rate exceeding the second threshold includes: the rate of increase of the data sequence value at the current time relative to the data sequence value at the earlier time exceeds the second threshold, and the rate of decrease of the data sequence value at the current time relative to the data sequence value at the later time exceeds the second threshold.

[0085] For example, a nearby moment can refer to another moment on the timeline that is adjacent to or separated from the current moment by a preset time window (predetermined time). A nearby moment can be a moment within the time period to be tested, or a moment outside the time period to be tested. A nearby moment can be a moment before the current moment, i.e., an earlier moment, or a moment after the current moment, i.e., a later moment. For example, a nearby moment can be a moment c-ε or a moment c+ε, where ε is a preset time window that can be flexibly set according to the business scenario.

[0086] In some embodiments, determining whether a candidate abnormal moment is a true "spurt" requires verifying whether its data has the characteristic of rising rapidly and then falling rapidly. This is achieved by comparing the data sequence value at the current moment with the data sequence value at a nearby moment. For example, comparing with the previous moment, the rate of increase is calculated, and comparing with the subsequent moment, the rate of decrease is calculated.

[0087] For example, the intermittent and discontinuous characteristics of burrs can be expressed as the following inequality.

[0088]

[0089] Where ywl(c) is the data sequence value at time c within the time period to be measured, ywl(c+ε1) is the data sequence value at the nearest time c+ε1, and ε1 represents the short time window between time c and time c. For definable coefficients, For comparison purposes, this inequality describes the process of burrs rapidly receding.

[0090]

[0091] Where ywl(c) is the data sequence value at time c within the time period to be measured, ywl(c-ε2) is the data sequence value at the nearest time c-ε2, and ε2 represents the short time window between time c and time c. For definable coefficients, For comparison purposes, this inequality describes the rapid rise of the burr. ε2 and ε1 can take the same value or different values.

[0092] The criteria can be determined based on factors such as sensitivity to sudden events, the basic fluctuation of business change rates, and alignment with the time window (scheduled time). For example, the more acute the spikes that need to be captured, the better. The smaller the value, the better; for example, if the business volume during normal periods fluctuates rapidly, The time window should be increased accordingly to avoid false alarms; for example, the smaller the time window, the more sensitive the rate of change is to instantaneous fluctuations. You can adjust it appropriately. Alternatively, you can use a large amount of training data to train the neural network model to obtain the desired result. .

[0093] The scale of the rate of change under normal conditions can be determined by the rate of change of data at each collection moment during the normal period (comparison period), the rate of change at the same moment during the normal period, or fixed empirical values. For example, the absolute value of the rate of change between each adjacent moment within the comparison period (the rate of change of the current data sequence value relative to the data sequence values ​​of its adjacent moments within the comparison period) can be calculated, and a statistical value (e.g., median, mean, quantile, etc.) can be taken as the statistical value. For example, for a rapid increase, the rate of change at time d relative to time (d-ε2) is used in the comparison period; for a rapid decrease, the rate of change at time d relative to time (d+ε1) is used in the comparison period. For example, in scenarios where the rate of change in business operations is relatively stable... It can be directly set to a fixed data value. It can also be obtained by training a neural network model using a large amount of training data. .

[0094] and The product of these values ​​is the threshold for identifying glitch moments. For example, if the rate of change of the current data sequence value relative to the data sequence values ​​of nearby moments is higher than this threshold, then the current moment can be determined as a glitch moment, and the data sequence value at the current moment is glitch data.

[0095] According to embodiments of this application, the intermittent characteristics of glitch are quantified by short-term change rate, and the time window is adjustable, thereby adapting to the response speed requirements of different business scenarios. By characterizing the complete waveform feature of glitch "rapidly rising and then rapidly falling back", unilateral jumps or drops are eliminated, improving the accuracy of glitch data identification.

[0096] Figure 3 A schematic diagram of burr data according to an embodiment of this application is shown.

[0097] like Figure 3 As shown, the horizontal axis represents the data acquisition time, and the vertical axis represents the data sequence value or the baseline data sequence value, such as traffic volume. Date 'a' represents the time period to be tested, and date 'b' represents the comparison time period.

[0098] For example, the comparison of the time distribution of business volume on date a and date b provides a quantitative description of the "glitch" phenomenon. As shown in the figure, the data distribution of date a and date b in the time intervals of 00:00-19:30 and 21:30-23:59 is basically the same, with the data values ​​changing smoothly. Compared with the data distribution of date b, date a shows a spike in data in the time interval of 19:30-21:30, exhibiting a glitch-like pattern. The typical manifestation of this type of abnormal data is a short-term spike with intermittent effects. From the chart, the difference between the normal business distribution and the glitch is very obvious. In terms of shape, these abnormal data resemble protruding burrs on the surface of clothing.

[0099] These types of "spicy" data occur in concentrated periods and in extremely high volumes. Unlike the smooth and continuous characteristics of business under normal conditions, they exhibit obvious intermittency and discontinuity. Therefore, it can be qualitatively concluded that spicy data is highly correlated with abnormal data, meaning that spicy data can well represent abnormal data.

[0100] According to an embodiment of this application, the data sequence value at the glitch moment is split into a data component and a glitch data component based on the baseline data sequence value at the comparison moment. The glitch data component is removed by: accumulating the glitch data component at each glitch moment within the test time period to obtain the total number of glitches; obtaining the total data volume based on the data sequence value collected within the test time period; and removing the total number of glitches from the total data volume.

[0101] For example, after constructing the spurious data feature, it is possible to identify abnormal and normal data. Spurious data can well represent abnormal data; after removing spurs, the remaining part is normal data. The data sequence value at the spurious moment can include normal data components and spurious data components. The normal data component is equivalent to the baseline data sequence value at the comparison moment. Subtracting the corresponding baseline data sequence value at the comparison moment from the data sequence value at the spurious moment yields the spurious data component at the spurious moment.

[0102] The glitch data processing method in this embodiment can accumulate the glitch data components at all glitch moments within the test period to obtain the total number of glitch data, thereby statistically analyzing the data anomalies within the test period. By accumulating the collected data sequence values ​​within the test period to obtain the total data volume, and then removing the obtained total number of glitch data from the total data volume, normal data within the test period can be obtained, thus achieving batch processing of anomalies and accelerating the processing speed of glitch data. For example, the data on date a (where glitch data exists) can be divided into two parts: a normal business part and an abnormal part. The abnormal part is the "area" occupied by the glitch data, and the rest is the normal part. By removing the normal data of date b from the data of date a, the abnormal part of the data on date a can be obtained. The selection of comparison date b will have some impact on the estimation of the abnormal data volume, but the overall impact is small.

[0103] The glitch data processing method in this application realizes the quantitative estimation of abnormal data volume, realizes the overall data restoration, and can output clean normal data volume that has eliminated the influence of all glitch, simplifying the calculation process and facilitating subsequent statistical analysis.

[0104] For example, the appearance of spikes reflects problems in the data recording or data acquisition process, and spikes can provide a basis for diagnosis in subsequent data acquisition stages.

[0105] According to an embodiment of this application, the glitch data processing method further includes: diagnosing the data acquisition source based on the determined glitch time, marking the data record, or reporting the abnormal process.

[0106] In some embodiments, the source of the glitch data is analyzed based on the determined glitch timing to determine whether the source, for example, has a fault, configuration error, performance bottleneck, or design flaw. The data acquisition source can be any link responsible for generating, recording, or transmitting raw data, such as event tracking code, log collector, sensor device, application programming interface, database connection, message queue, or reporting gateway.

[0107] For example, diagnostic methods may include: checking the consistency of timestamps, data format, recording frequency, reporting success rate, network latency, cache queue status, etc. of the data collection source, and performing cross-validation in combination with the time pattern of spikes (e.g., whether they are concentrated in a specific data collection source or whether they occur periodically).

[0108] The glitch data processing method in this embodiment can also distinguish whether the anomaly occurs during the data recording stage or the data reporting stage, and label the corresponding anomaly link to facilitate subsequent manual or automatic repair. Data recording anomalies refer to problems that occur during data generation or local persistence. Data reporting anomalies refer to problems that occur during data transmission from the acquisition end to the server end. The system can output a diagnostic report, specifying "acquisition source identifier + anomaly type (recording anomaly or reporting anomaly) + occurrence time + evidence (glitch morphology)," and store the diagnostic report in the database or trigger an alarm.

[0109] According to embodiments of this application, the mechanism for diagnosing data acquisition sources achieves a closed loop from data quality detection to root cause localization, reducing investigation time and costs. This diagnostic mechanism can trigger subsequent remedial actions, reducing the frequency of glitches in subsequent data sequences, improving data acquisition quality, and enhancing data reliability.

[0110] This application identifies abnormal data at each moment in normal data statistics, provides a basis for identification, removes abnormal data, and performs data restoration. It identifies abnormal data through glitch detection, thereby tracing the source of the glitch and improving data acquisition quality. To promptly detect data anomalies, the method provided in this embodiment can also rely on glitch features to monitor the data sequence.

[0111] According to an embodiment of this application, the glitch data processing method further includes: establishing real-time monitoring to acquire data sequence values ​​at a minute-level frequency; and issuing an early warning signal in response to the detection of a glitch.

[0112] For example, by combining a mathematical description of the spike data (brief spikes, intermittent occurrences, discontinuities) and based on the frequency of monitoring statistics (typically on the minute level, for example, it can be set to 1 to 3 minutes), a threshold monitoring for spike detection can be set. The spike detection threshold monitoring could be, for example, triggering an alert when the data sequence value at any acquisition moment within the test period exceeds the set monitoring threshold. The monitoring threshold can be a safe range for the data sequence values, and its value can be determined through statistical analysis of a large amount of historical data.

[0113] According to the embodiments of this application, by establishing a glitch monitoring mechanism, the timeliness of anomaly identification is improved, enabling early detection and early processing, and reducing the negative impact of abnormal data.

[0114] According to an embodiment of this application, the burr data processing method further includes: extracting the data curve graph within the time period to be tested, comparing it with a pre-stored historical burr graph template; and determining the time period to be tested as having burr anomalies in response to the similarity comparison result exceeding a preset threshold.

[0115] For example, visualizing data is an intuitive and feasible method. Graphical methods can visually identify spikes and clearly determine the magnitude of abnormal data from the area of ​​the graph, thereby enabling the identification and calculation of abnormal data.

[0116] For example, the data sequence values ​​(numerical values ​​arranged on a time scale) within the test time period are transformed into a visualized curve. In actual computer processing, it is not necessarily necessary to actually generate an image, but rather to extract the shape features of the curve (e.g., peak height, width, rising slope, falling slope, waveform symmetry, etc.) to form a feature vector or waveform data that can characterize the curve's shape. This provides a graphical comparison object for subsequent similarity comparisons, rather than relying solely on numerical thresholds.

[0117] The system pre-collects typical data curves that have been identified as glitch in history and stores them as templates in a database or template library. Each template records the waveform characteristics of that glitch (e.g., the shape, duration, and peak-to-base ratio of a rapid surge followed by a rapid decline). Templates can be manually annotated historical glitch samples or glitch cases automatically accumulated by the system and confirmed through numerical methods.

[0118] The data curve graph within the test time period is matched with each historical spike graph template to calculate a similarity score (e.g., between 0 and 1, with 1 indicating complete similarity). Similarity algorithms can employ dynamic time warping, cosine similarity, Pearson correlation coefficient, or deep learning-based feature matching. This is a pattern recognition or shape matching process that does not rely on absolute numerical values ​​but focuses on the relative shape of the curves.

[0119] For example, if the similarity comparison result exceeds a preset threshold, it is determined that a glitch has occurred. For instance, when the similarity between the graphic curve corresponding to the time period to be tested and a certain historical glitch template reaches or exceeds a set graphic threshold (such as 80%), it is considered that there is an anomaly in the time period to be tested that highly matches the typical glitch pattern, and the moment corresponding to the typical glitch is determined as the glitch moment.

[0120] The preset threshold (the set image threshold) controls the sensitivity of the recognition. The lower the threshold, the easier it is to detect (which may increase false alarms); the higher the threshold, the stricter the detection conditions (which may increase false negatives).

[0121] For example, the glitch data processing method provided in this embodiment utilizes a graphic recognition template function, providing historically occurring glitch graphics as templates for system recognition and matching. A certain similarity percentage is used as a monitoring threshold; when the similarity exceeds the threshold, it is determined that a glitch has occurred. The application of the graphical method compensates for the shortcomings of purely numerical threshold determination, reduces the difficulty of threshold parameter tuning, and avoids repeated adjustments for different business scenarios. , , , Parameters such as these can effectively identify glitches with typical waveform characteristics but insignificant numerical amplitudes, improving the robustness and interpretability of the detection.

[0122] The glitch data processing method provided in this application is applicable to various business scenarios and has clear business significance, especially suitable for businesses with large data volumes, wide coverage, and a large number of customers. The method provided in this application can quantify the entire detection process, quantify data differences, reduce the negative impact of abnormal data on business, enhance the value of business data, and maintain the reference value of data analysis.

[0123] Based on the above-described burr data processing method, embodiments of this application also provide a burr data processing apparatus. The following will be combined with... Figure 4 The device is described in detail.

[0124] Figure 4 A schematic block diagram of a burr data processing apparatus according to an embodiment of this application is shown.

[0125] like Figure 4 As shown, the burr data processing device 400 of this embodiment includes a test data acquisition module 410, a reference data acquisition module 420, a candidate anomaly determination module 430, a rate of change calculation module 440, and a burr data removal module 450.

[0126] The test data acquisition module 410 is used to acquire data sequence values ​​collected according to the time scale within the test time period. In one embodiment, the test data acquisition module 410 can be used to execute step S210 described above, which will not be repeated here.

[0127] The benchmark data acquisition module 420 is used to acquire benchmark data sequence values ​​collected according to the time scale within the comparison time period. In one embodiment, the benchmark data acquisition module 420 can be used to perform step S220 described above, which will not be repeated here.

[0128] The candidate anomaly determination module 430 is used to mark the current time as a candidate anomaly time in response to a positive deviation between the data sequence value at the current time and the baseline data sequence value at the comparison time exceeding a first threshold. Here, the current time is selected from the time period to be tested, and the comparison time is the time corresponding to the current time within the comparison time period. In one embodiment, the candidate anomaly determination module 430 can be used to execute step S230 described above, which will not be repeated here.

[0129] The rate of change calculation module 440 is used to calculate the data change rate at the current moment. In response to the data change rate exceeding a second threshold, the candidate abnormal moment is determined as a spike moment. In one embodiment, the rate of change calculation module 440 can be used to execute step S240 described above, which will not be repeated here.

[0130] The glitch data removal module 450 is used to split the data sequence value at the glitch time into a data component and a glitch data component based on the baseline data sequence value at the comparison time, and remove the glitch data component. In one embodiment, the glitch data removal module 450 can be used to perform step S250 described above, which will not be repeated here.

[0131] According to an embodiment of this application, the comparison time period is a historical time period without spikes. The comparison time period includes a single historical time period that is an integer number of statistical periods apart from the time period to be tested, or the average of N single historical time periods, where N is an integer greater than 1.

[0132] According to an embodiment of this application, the rate of change calculation module 440 is further configured to: calculate the rate of change of the data sequence value at the current time relative to the data sequence value at an adjacent time, wherein the adjacent time is a time interval of a predetermined time from the current time.

[0133] According to an embodiment of this application, the adjacent time includes an earlier time and a later time, the earlier time is earlier than the current time, and the later time is later than the current time. The data change rate exceeding the second threshold includes: the rate of increase of the data sequence value at the current time relative to the data sequence value at the earlier time exceeds the second threshold, and the rate of decrease of the data sequence value at the current time relative to the data sequence value at the later time exceeds the second threshold.

[0134] According to an embodiment of this application, the first threshold is a first fixed value, or is calculated based on the benchmark data sequence value of the comparison time period; the second threshold is a second fixed value, or is calculated based on the benchmark data sequence value of the comparison time period.

[0135] According to an embodiment of this application, the glitch data removal module 450 is further configured to: accumulate the glitch data components at each glitch moment within the test time period to obtain the total number of glitch data; obtain the total data volume based on the data sequence values ​​collected within the test time period; and remove the total number of glitch data from the total data volume.

[0136] According to an embodiment of this application, the burr data processing device 400 further includes a burr data diagnosis module 460, which is used to: diagnose the data acquisition source based on the determined burr time, mark the data record, or report the abnormal link.

[0137] According to an embodiment of this application, the burr data processing device 400 further includes a graphic template comparison module 470, which is used to: extract the data curve graphic within the time period to be tested, compare it with the pre-stored historical burr graphic template; and determine the time period to be tested as having burr anomalies in response to the similarity comparison result exceeding a preset threshold.

[0138] According to embodiments of this application, any multiple modules among the test data acquisition module 410, reference data acquisition module 420, candidate anomaly determination module 430, rate of change calculation module 440, and glitch data removal module 450 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this application, at least one of the test data acquisition module 410, reference data acquisition module 420, candidate anomaly determination module 430, rate of change calculation module 440, and glitch data removal module 450 can be at least partially implemented as hardware circuits, such as field-programmable gate arrays, programmable logic arrays, systems-on-a-chip, systems-on-a-substrate, systems-on-package, application-specific integrated circuits, or any other reasonable means of integrating or packaging circuits, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods. Alternatively, at least one of the following modules can be implemented, at least partially, as a computer program module: the test data acquisition module 410, the baseline data acquisition module 420, the candidate anomaly determination module 430, the rate of change calculation module 440, and the spur data removal module 450. When the computer program module is run, it can perform the corresponding function.

[0139] Figure 5 A block diagram schematically illustrates an electronic device suitable for implementing a glitch data processing method according to an embodiment of this application.

[0140] like Figure 5As shown, an electronic device 1200 according to an embodiment of this application includes a processor 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory 1202 or a program loaded from a storage portion 1208 into a random access memory 1203. The processor 1201 may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a dedicated microprocessor. The processor 1201 may also include onboard memory for caching purposes. The processor 1201 may include a single processing unit or multiple processing units for executing different steps of the method flow according to an embodiment of this application.

[0141] Random access memory 1203 stores various programs and data required for the operation of electronic device 1200. Processor 1201, read-only memory 1202, and random access memory 1203 are interconnected via bus 1204. Processor 1201 executes various steps of the method flow according to embodiments of this application by executing programs in read-only memory 1202 and / or random access memory 1203. It should be noted that the programs may also be stored in one or more memories other than read-only memory 1202 and random access memory 1203. Processor 1201 may also execute various steps of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0142] According to embodiments of this application, the electronic device 1200 may further include an input / output interface 1205, which is also connected to the bus 1204. The electronic device 1200 may also include one or more of the following components connected to the input / output interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including a cathode ray tube, liquid crystal display, etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card, such as a local area network card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the input / output interface 1205 as needed. A removable medium 1211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1210 as needed so that computer programs read from it can be installed into the storage section 1208 as needed.

[0143] Embodiments of this application also provide a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0144] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. In embodiments of this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include the read-only memory 1202, and / or random access memory 1203, and / or one or more memories other than read-only memory 1202 and random access memory 1203 described above.

[0145] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this application.

[0146] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1209, and / or installed from the removable medium 1211. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0147] In embodiments of this application, the computer program can be downloaded and installed from a network via communication section 1209, and / or installed from removable medium 1211. When the computer program is executed by processor 1201, it performs the functions defined in the system of this application embodiment. According to embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0148] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0149] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0150] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A method for processing glitch data, characterized in that, The method includes: Obtain the data sequence values ​​collected according to the time scale within the time period to be tested; Obtain the baseline data sequence values ​​collected according to the time scale within the comparison period; In response to the positive deviation between the data sequence value at the current time and the baseline data sequence value at the comparison time exceeding a first threshold, the current time is marked as a candidate abnormal time, wherein the current time is selected from the time period to be tested, and the comparison time is the time in the comparison time period that corresponds to the current time; Calculate the data change rate at the current moment, and in response to the data change rate exceeding a second threshold, determine the candidate abnormal moment as a glitch moment; and Based on the baseline data sequence value at the comparison time, the data sequence value at the glitch time is split into a data component and a glitch data component, and the glitch data component is removed.

2. The method according to claim 1, characterized in that, The comparison time period is a historical period of time without burrs. The comparison time period includes a single historical time period that is an integer number of statistical periods apart from the time period to be tested, or the average of N single historical time periods, where N is an integer greater than 1.

3. The method according to claim 1, characterized in that, The calculation of the data change rate at the current moment includes: Calculate the rate of change of the data sequence value at the current moment relative to the data sequence value at an adjacent moment, where the adjacent moment is a moment at a predetermined time interval from the current moment.

4. The method according to claim 3, characterized in that, The nearest time includes an earlier time and a later time, where the earlier time is earlier than the current time and the later time is later than the current time. The data change rate exceeding the second threshold includes: The rate of increase of the data sequence value at the current moment relative to the data sequence value at the previous moment exceeds the second threshold, and the rate of decrease of the data sequence value at the current moment relative to the data sequence value at the subsequent moment exceeds the second threshold.

5. The method according to claim 3, characterized in that, The first threshold is a first fixed value, or is calculated based on the benchmark data sequence value of the comparison time period; The second threshold is a second fixed value, or it is calculated based on the benchmark data sequence value of the comparison time period.

6. The method according to claim 1, characterized in that, The reference data sequence value based on the comparison time is used to split the data sequence value at the glitch time into a data component and a glitch data component. Removing the glitch data component includes: The puncture data components at each puncture moment within the time period to be tested are accumulated to obtain the total number of punctures; The total data volume is obtained based on the data sequence values ​​collected within the test time period; and The total number of burrs is removed from the total data volume.

7. The method according to claim 1, characterized in that, The method further includes: Based on the determined glitch moment, the data acquisition source is diagnosed, and the data record or abnormal link is marked.

8. The method according to claim 1, characterized in that, The method further includes: Extract the data curve graph within the time period to be tested, and compare its similarity with a pre-stored historical spike graph template; and If the similarity comparison result exceeds a preset threshold, the time period to be tested is determined to have an abnormal burr.

9. A glitch data processing device, characterized in that, The device includes: The test data acquisition module is used to acquire data sequence values ​​collected according to the time scale within the test time period; The benchmark data acquisition module is used to acquire benchmark data sequence values ​​collected according to the time scale within the comparison time period; The candidate anomaly determination module is used to mark the current time as a candidate anomaly time in response to the positive deviation between the data sequence value at the current time and the baseline data sequence value at the comparison time exceeding a first threshold. The current time is selected from the time period to be tested, and the comparison time is the time in the comparison time period that corresponds to the current time. The rate of change calculation module is used to calculate the data change rate at the current moment, and in response to the data change rate exceeding a second threshold, to determine the candidate abnormal moment as a spike moment; and The glitch data removal module is used to split the data sequence value at the glitch time into a data component and a glitch data component based on the baseline data sequence value at the comparison time, and remove the glitch data component.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

Images