Abnormal data identification method and device, electronic equipment, medium and program product

Abstract

The invention provides an abnormal data identification method and device, electronic equipment, a medium and a program product, and relates to the technical field of data processing, and the method comprises the steps: carrying out the fluctuation trend alignment of a reference data curve and a to-be-identified data curve, enabling the fluctuations of the two curves to be corresponding, and acquiring a fluctuation similarity score between the fluctuation information of the reference data curve and the waveform information of the to-be-identified data curve aligned with the reference data curve, determining that the to-be-identified data curve is an abnormal data curve when the fluctuation similarity score is greater than a first preset score, and determining that the to-be-identified data curve is an abnormal data curve after the two curves are aligned. And then similarity comparison is carried out to calibrate hysteresis of any curve caused by time, so that hysteresis interference is avoided, and the accuracy of abnormal data identification is improved.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to an abnormal data identification method, apparatus, electronic device, medium, and program product. Background Technology

[0002] Data records the results of business processes, such as sales revenue, user volume, production cost increases, and number of failures. The value of analyzing this type of data for enterprises is self-evident. Identifying outliers within this data is crucial, and how to identify outliers has become a pressing issue. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this disclosure provides an abnormal data identification method, apparatus, electronic device, medium, and program product.

[0004] According to a first aspect of the present disclosure, an abnormal data identification method is provided, comprising: aligning the fluctuation trends of a reference data curve and a data curve to be identified, wherein the reference data curve is a normal data curve; acquiring a fluctuation similarity score between fluctuation information of the reference data curve and waveform information of the data curve to be identified after alignment with the reference data curve, wherein the fluctuation information includes increase information and / or upward trend information, and the fluctuation similarity score is inversely correlated with the degree of similarity; and determining the data curve to be identified as an abnormal data curve if the fluctuation similarity score is greater than a first preset score. Thus, by aligning the two curves, the hysteresis of the curves is calibrated, hysteresis interference is avoided, and the accuracy of abnormal data identification is improved.

[0005] In some possible implementations, the baseline data curve includes multiple normal data points, and the data curve to be identified includes multiple data points to be identified. The baseline data curve and the data curve to be identified are aligned by: determining the similarity distance between the fluctuation information of each data point on the baseline data curve and the fluctuation information of each data point on the data curve to be identified, thus obtaining a distance matrix; and determining the correspondence between the data points on the baseline data curve and the data points to be identified on the data curve to be identified based on the distance matrix. This aligns the two curves, facilitating subsequent trend analysis.

[0006] In some possible implementations, determining the correspondence between data points on the reference data curve and data points on the data curve to be identified, based on the distance matrix, includes: traversing the distance matrix in a preset manner to determine the path with the minimum sum of similar distances; and determining the correspondence between data points on the reference data curve and data points on the data curve to be identified based on the path. Thus, by using the path with the minimum sum of similar distances, the trends of the two curves are aligned as much as possible, facilitating subsequent comparison using data points with the same trend and improving the accuracy of anomaly identification.

[0007] In some possible implementations, the distance matrix is ​​an m-row, n-column matrix. The step of traversing the distance matrix according to a preset method to determine the path with the minimum sum of similar distances includes: traversing from the first element of the distance matrix to the second element to determine the path with the minimum sum of similar distances, wherein the first element is located in the first row and first column of the distance matrix, and the second element is located in the m-th row and n-th column of the distance matrix; or the first element is located in the m-th row and first column of the distance matrix, and the second element is located in the first row and n-th column of the distance matrix. This facilitates subsequent comparison using data points with the same trend, improving the accuracy of anomaly identification.

[0008] In some possible implementations, obtaining a fluctuation similarity score between the fluctuation information of a baseline data curve and the fluctuation information of a target data curve aligned with the baseline data curve includes: obtaining a fluctuation similarity sub-score between the fluctuation information of a data point on the baseline data curve and the fluctuation information of a target data point aligned with that data point, based on the correspondence, to obtain multiple fluctuation similarity sub-scores; and obtaining the fluctuation similarity score based on the multiple fluctuation similarity sub-scores. Therefore, the fluctuation similarity score can take into account the fluctuation factors of more points, making it more reflective of the fluctuation differences between the two curves, further improving the accuracy of abnormal data identification.

[0009] In some possible implementations, the method further includes: identifying data points whose fluctuation similarity sub-scores are greater than a second preset score from the plurality of fluctuation similarity sub-scores as anomalous data points. This identifies specific anomalous data points, facilitating accurate anomaly alerts.

[0010] In some possible implementations, obtaining a fluctuation similarity score between the fluctuation information of a reference data curve and the waveform information of the data curve to be identified aligned with the reference data curve includes: dividing the reference data curve into multiple first sub-curves, and dividing the data curve to be identified aligned with the reference data curve into multiple second sub-curves; obtaining a fluctuation similarity sub-score between the fluctuation information of each first sub-curve and the fluctuation information of the second sub-curve aligned with the first sub-curve, thus obtaining multiple fluctuation similarity sub-scores; and obtaining the fluctuation similarity score based on the multiple fluctuation similarity sub-scores. Compared to the aforementioned point-by-point comparison method, this improves the speed of obtaining the fluctuation similarity score.

[0011] In some possible implementations, the fluctuation information includes both price increase information and upward trend information. A fluctuation similarity sub-score is obtained by: obtaining the upward trend similarity score between the upward trend information of each first sub-curve and the upward trend information of the corresponding second sub-curve; obtaining the price increase similarity score between the price increase information of each first sub-curve and the price increase information of the corresponding second sub-curve; and summing the upward trend similarity score and the price increase similarity score to obtain the fluctuation similarity sub-score. Price increase information reflects local fluctuations, while upward trend information reflects overall fluctuations. Combining these two types of information to obtain the fluctuation similarity sub-score, and using this score for anomaly data identification, further improves the accuracy of anomaly identification.

[0012] In some possible implementations, the method further includes: obtaining an overall similarity score between the baseline data curve and the data curve to be identified; obtaining a similarity score based on the sum of the overall similarity score and the fluctuation similarity score; and determining the data curve to be identified as an abnormal data curve if the similarity score is greater than a third preset score. Thus, the overall similarity score reflects the overall trend, and the fluctuation similarity score reflects the local trend. The resulting similarity score reflects both the overall trend and the local trend of the curve. Using the similarity score for abnormal data identification further improves the accuracy of anomaly identification.

[0013] According to a second aspect of the present disclosure, an abnormal data identification device is provided, comprising: The alignment module is configured to align the fluctuation trends of a reference data curve and a data curve to be identified, wherein the reference data curve is a normal data curve; The acquisition module is configured to acquire a fluctuation similarity score between fluctuation information of a baseline data curve and waveform information of a data curve to be identified after alignment with the baseline data curve, wherein the fluctuation information includes increase information and / or upward trend information, and the fluctuation similarity score is inversely correlated with the degree of similarity; The identification module is configured to determine that the data curve to be identified is an abnormal data curve when the fluctuation similarity score is greater than a first preset score.

[0014] In one possible implementation, the alignment module includes: The matrix determination module is configured to determine the similarity distance between the fluctuation information of each data point on the baseline data curve and the fluctuation information of each data point to be identified on the data curve to be identified, thereby obtaining a distance matrix; The correspondence determination module is configured to determine the correspondence between data points on the reference data curve and data points to be identified on the data curve to be identified, based on the distance matrix.

[0015] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the steps of the method described in the first aspect when executing the instructions.

[0016] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the steps of the method provided in the first aspect of the present disclosure.

[0017] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0020] Figure 1 This is a schematic diagram of the baseline data curve and the data curve to be identified.

[0021] Figure 2 This is a flowchart illustrating an abnormal data identification method according to an exemplary embodiment.

[0022] Figure 3 This is a flowchart illustrating an abnormal data identification method according to another exemplary embodiment.

[0023] Figure 4 This is a schematic diagram of the initial distance matrix shown in an exemplary embodiment.

[0024] Figure 5 This is a schematic diagram of a distance matrix shown in an exemplary embodiment.

[0025] Figure 6 This is a schematic diagram of a distance matrix shown in another exemplary embodiment.

[0026] Figure 7 This is an exemplary embodiment illustrating the alignment of a reference data curve and a data curve to be identified.

[0027] Figure 8 This is a flowchart illustrating an abnormal data identification method according to another exemplary embodiment.

[0028] Figure 9 This is a block diagram illustrating an abnormal data identification device according to an exemplary embodiment.

[0029] Figure 10 This is a block diagram illustrating an electronic device for an abnormal data identification method according to an exemplary embodiment. Detailed Implementation

[0030] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0031] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0032] Data records the results of business processes, such as sales revenue, user volume, production costs, and number of failures. The value of analyzing this type of data for businesses is self-evident. For example, in sales analysis, different data curves are generated for different sales channels. It's necessary to identify anomalous data curves whose trends don't align with the overall sales trend from among these multiple curves. Another example is that, under normal circumstances, product sales volume and sales revenue trends are consistent. If product sales volume increases but sales revenue doesn't increase proportionally, it's necessary to analyze the data to determine the cause of the anomaly and mitigate losses. Therefore, analyzing outliers in the data is crucial, and identifying these outliers is a pressing issue that needs to be addressed.

[0033] To identify outlier data, a baseline data curve and a data curve to be identified are determined. Then, data points at the same time points on both curves are compared. Based on the differences in these data points, it is determined whether the data curve to be identified is an outlier. If it is, the location of the outlier points within that curve is determined. For example, combining... Figure 1 The baseline data curve includes data points A1 to A6, and the data curve to be identified includes data points B1 to B6. The difference between data point A1 on the baseline curve and data point B1 on the data curve to be identified is obtained. Similarly, the difference between data point A2 on the baseline curve and data point B2 on the data curve to be identified is obtained, until the difference values ​​of all data points at the same time on both curves are compared, resulting in six difference values. These six difference values ​​are analyzed to determine whether the data curve to be identified is an anomalous curve, and the location of anomalous data points on the anomalous curve.

[0034] To facilitate understanding, let's consider a specific scenario: A product has a promotional event on channel A on Monday, and sales on channel A begin to rise on Tuesday. On channel B, the promotional event begins on Wednesday, and sales on channel B begin to rise on Thursday. For example, the baseline curve is the total sales curve of the product across all channels, and the data curve to be identified is the sales data curve of the product on channel B. If we use... Figure 1 The method involves comparing the differences between two curves point-by-point according to the time series. The baseline curve rose on Tuesday, but the data curve to be identified did not rise on Tuesday. Figure 1 The method described above might misclassify the data curve to be identified as an anomaly. According to the information provided, the data curve to be identified is not actually an anomaly; the delay in the activity on channel B causes the data fluctuations on the curve to be identified to be delayed. Therefore, it is clear that the accuracy of anomaly identification using the above method is not high.

[0035] To address the aforementioned issues, this disclosure provides a method for identifying abnormal data. Please refer to [link / reference]. Figure 1 The abnormal data identification method can be applied to Figure 9 The abnormal data identification device 200 shown Figure 10 The illustrated electronic device 800, computer program product, and computer-readable storage medium are described below. Figure 2 The process shown will be described in detail. The abnormal data identification method may include the following steps: Step S110: Align the fluctuation trends of the baseline data curve and the data curve to be identified.

[0036] Both the baseline data curve and the data curve to be identified can be time curves.

[0037] The reference data curve is a normal data curve. For example, the reference data curve can be a normal data curve detected by the user or predetermined. The reference curve can also be a data curve identified as meeting expectations or set targets by the abnormal data identification method provided in this disclosure. There can be one or more reference data curves.

[0038] The number of data curves to be identified can be determined based on the business scenario; there can be one or multiple curves. For example, the data curves to be identified could be sales data from different sales channels, in which case the number of curves is determined by the number of sales channels. As another example, the data curves to be identified could be sales data from different mobile phone models, in which case the number of curves is determined by the number of mobile phone models.

[0039] Aligning the baseline data curve and the data curve to be identified by their fluctuation trends aligns the two curves in terms of both fluctuation trend and shape, ensuring that their fluctuation trends correspond and resolving delays caused by differences in time, channel, etc. After alignment, points on the baseline data curve naturally align with points on the data curve to be identified. This alignment is reflected in the fact that the peaks and troughs of the two curves are aligned.

[0040] Step S120: Obtain the fluctuation similarity score between the fluctuation information of the reference data curve and the waveform information of the data curve to be identified after being aligned with the reference data curve.

[0041] The fluctuation information includes price increase information and / or upward trend information, and the fluctuation information can be numerical. The fluctuation similarity score is inversely correlated with the degree of similarity.

[0042] Optionally, the baseline data curve consists of multiple normal data points. A data point on the baseline data curve is selected, and its fluctuation information is determined by comparing it with data points from previous time points. The fluctuation information of the baseline data curve can be obtained based on the fluctuation information of the data point. Since the baseline data curve is composed of multiple data points, a preset number of data points can be selected, and their fluctuation information can be used to reflect the fluctuation information of the baseline data curve. For example, the average of the fluctuation information of the preset number of data points can be used as the fluctuation information of the baseline data curve. The preset number is less than or equal to the total number of data points in the baseline data curve.

[0043] Similarly, the data curve to be identified consists of multiple data points. A corresponding data point is identified on the curve that aligns with the identified data point. This data point is then compared to previous data points to determine its fluctuation information. Based on the fluctuation information of these data points, the fluctuation information of the data curve to be identified can be obtained. With a preset number of selected data points, the fluctuation information of multiple data points is obtained, and the average of these fluctuation information is used as the fluctuation information of the data curve to be identified.

[0044] Step S130: If the fluctuation similarity score is greater than the first preset score, the data curve to be identified is determined to be an abnormal data curve.

[0045] A first preset score is set. If the fluctuation similarity score is greater than the first preset score, the data curve to be identified is determined to be an abnormal data curve. Conversely, if the fluctuation similarity score is less than or equal to the first preset score, the data curve to be identified is determined to be a normal data curve.

[0046] In this embodiment, the fluctuation trends of the reference data curve and the data curve to be identified are aligned so that the fluctuations of the two curves correspond. Then, the fluctuation similarity score between the fluctuation information of the reference data curve and the waveform information of the data curve to be identified after alignment with the reference data curve is obtained. If the fluctuation similarity score is greater than the first preset score, the data curve to be identified is determined to be an abnormal data curve. After the two curves are aligned, a similarity comparison is performed to calibrate the lag caused by time in either curve, so as to avoid lag interference, thereby improving the accuracy of abnormal data identification.

[0047] Optionally, abnormal data curves can be marked to facilitate alerts and reduce business losses. For example, when abnormal data curves are displayed on the screen of an electronic device, they can be displayed differently from normal data curves. For instance, normal data curves can be displayed as solid lines, while abnormal data curves are displayed as dashed lines. Alternatively, abnormal and normal data curves can be displayed in different colors.

[0048] The baseline data curve includes multiple normal data points, and the data curve to be identified includes multiple data points to be identified. For an example, please refer to [link to example]. Figure 3 The reference data curve and the data curve to be identified are aligned in the following manner: Step S111: Determine the similarity distance between the fluctuation information of each data point on the baseline data curve and the fluctuation information of each data point to be identified on the data curve to be identified, and obtain the distance matrix.

[0049] The similarity distance characterizes the degree of similarity between the fluctuation information of the data point and the fluctuation information of the data point to be identified that is aligned with the data point.

[0050] Optionally, the distance matrix is ​​initialized first. Initializing the distance matrix can also be understood as the process of constructing an initial distance matrix between the reference data curve and the data curve to be identified. For example, based on the number of data points m on the reference data curve and the number of data points n on the data curve to be identified, an initial distance matrix of m rows and n columns is constructed. Continuing with... Figure 1 Taking the baseline data curve and the data curve to be identified as examples, the constructed 6×6 initial distance matrix is ​​as follows: Figure 4 As shown. Then, a similarity distance is calculated between every two data points and the data points to be identified, and this distance is used as an element of the distance matrix. After calculating the similarity distance between each data point and all the data points to be identified, m·n elements are obtained. Finally, the m·n elements are used to fill the initial distance matrix in order to obtain the distance matrix. Continuing with the above example, 36 similarity distances are used to fill the matrix. Figure 4 The initial distance matrix in the matrix is ​​obtained. Figure 5 The distance matrix is ​​shown. Please refer to [link / reference]. Figure 5 In the distance matrix, the element 1.6 in the first row and first column represents the similarity distance between data point A1 and the data point B1 to be identified. The element 1.5 in the first row and fifth column represents the similarity distance between data point A5 and the data point B1 to be identified.

[0051] In one implementation, the fluctuation information includes both price increase information and upward trend information. Upward trend information reflects the overall trend of the curve, while price increase information reflects local changes in the curve. This implementation considers both local and overall factors, determining a first similarity distance between the price increase information of each data point on the reference data curve and the price increase information of each data point on the data curve to be identified, and determining a second similarity distance between the upward trend information of each data point on the reference data curve and the upward trend information of each data point on the data curve to be identified. The total similarity distance is obtained by summing the first and second similarity distances.

[0052] For example, the waveform information of each data point on the baseline data curve is determined. The fluctuation information of the baseline data curve is represented by (increase information, increase information). If the data point does not increase, the increase information is 0; if the increase indicates growth, the increase information is 1; and if the increase indicates decrease, the increase information is -1. The increase information is obtained by subtracting the previous data point from the current data point and then using the quotient between this difference and the previous data point.

[0053] Combination Figure 1For data point A1 on the baseline data curve, there are no other data points before A1. Therefore, the fluctuation information of data point A1 can be consistent with the fluctuation information of data point A2 that follows it. Data point A2 has no fluctuation compared to data point A1, and the fluctuation information of data point A2 is (0, 0.0). Therefore, the fluctuation information of data point A1 is (0, 0.0). Data point A3 shows an increasing trend compared to data point A2, with an increase of 200%. Therefore, the fluctuation information of data point A3 is (1, 2.0). Similarly, the fluctuation information of data point A4 is calculated to be (0, 0.0). The fluctuation information of data point A5 is calculated to be (-1, -0.333). The fluctuation information of data point A6 is calculated to be (1, 1.0).

[0054] Similarly, targeting Figure 1 The fluctuation information for the data curves to be identified is as follows: Data point B2: (1, 2.0); Data point B1: (1, 2.0); Data point B3: (-1, -0.333); Data point B4: (0, 0); Data point B5: (1, 1.0); Data point B6: (0, 0).

[0055] After determining the fluctuation information of each point, the upward trend information of the data point and its corresponding data point to be identified is compared according to the correspondence. If the upward trend information of the two is consistent, the first similarity distance is determined to be 0; otherwise, if they are inconsistent, the first similarity distance is determined to be 1. Then, the difference between the two increase information is determined, and the second similarity distance is determined based on the difference.

[0056] For example, the second similarity distance can be determined as follows: a preset mapping relationship is established between each of multiple difference ranges and the second similarity distance. The range in which the difference lies is determined, and then the second similarity distance corresponding to that range is determined according to the preset mapping relationship. For example, if the difference range is 0, the second similarity distance is 0. The difference range is (0, 0.2], and the corresponding second similarity distance is 0.1. The difference range is (0.2, 0.4], and the corresponding second similarity distance is 0.2. The difference range is (0.4, 0.6], and the corresponding second similarity distance is 0.3. The difference range is (0.6, 0.8], and the corresponding second similarity distance is 0.4. The difference range is (0.5, 1.0], and the corresponding second similarity distance is 0.5. The difference range is (1, 2.0], and the corresponding second similarity distance is 0.6. The difference range is (2.0, 3.0], and the corresponding second similarity distance is 0.7. The difference range is (3.0, 5.0], and the corresponding second similarity distance is 0.8. The difference range is (5.0, +∞), and the corresponding second similarity distance is 0.9.

[0057] The similarity distance is obtained by summing the first and second similarity distances. Continuing with the example above, considering the fluctuation information (0, 0.0) of data point A1 and the fluctuation information (1, 2.0) of the data point to be identified B1, the upward trend information 0 of data point A1 and the upward trend information 1 of the data point to be identified are not equal, so the first similarity distance is 1. The difference between the upward trend information 0.0 of data point A1 and the upward trend information 2.0 of the data point to be identified B1 is 2.0, which falls within the difference range (1, 2.0]. The second similarity distance corresponding to this difference range is 0.6. Therefore, the similarity distance of 1.6 between the fluctuation information of data point A1 and the fluctuation information of the data point to be identified B1 is calculated.

[0058] Similarly, other similarity distances can be calculated using the above method. For example, the similarity distance between the fluctuation information of data point A2 and the fluctuation information of the data point B1 to be identified is 1.6.

[0059] Step S112: Determine the correspondence between the data points on the reference data curve and the data points to be identified on the data curve to be identified based on the distance matrix.

[0060] Based on the distance matrix, the relationship path is determined by traversing the distance matrix according to a preset method. This relationship path is composed of elements from the distance matrix; that is, the relationship path is formed by... Figure 5 The distance matrix is ​​composed of adjacent cells, thus the relational path constrains the matching of certain data points on the baseline data curve with the data points to be identified on the data curve to be identified. Based on the relational path, the correspondence between the data points on the baseline data curve and the data points to be identified on the data curve to be identified is determined.

[0061] There can be multiple determined relationship paths. One approach is to calculate the sum of similarity distances for each relationship path, obtaining a distance sum. From these multiple distance sums, a relationship path is determined corresponding to any distance sum smaller than a preset distance. Based on this relationship path, the correspondence between data points on the baseline data curve and data points on the data curve to be identified is determined.

[0062] As another approach, the correspondence is determined by traversing the distance matrix in a preset manner to find the path with the smallest sum of similar distances; and based on the path, determining the correspondence between the data points on the baseline data curve and the data points to be identified on the data curve to be identified.

[0063] Based on the examples above, traverse Figure 5 The distance matrix shown indicates that the path with the minimum sum of similar distances is as follows: Figure 6 The path is shown within the dashed box. Figure 6The path in the distance matrix includes the similarity distance of 1.6 in the first row and first column. (Refer to...) Figure 7 The correspondence includes the correspondence between data point A1 and the data point to be identified, B1. Figure 6 The path includes a similarity distance of 1.6 in the first row and second column of the distance matrix. Continue referring to... Figure 7 The correspondence includes the correspondence between data point A2 and the data point to be identified B1, and is drawn in this way. Figure 7 The corresponding relationship curve is shown.

[0064] In this approach, the path with the minimum sum of similar distances is determined, and the cumulative cost on the path with the minimum similarity distance is minimized. The correspondence determined according to this path shows a higher degree of fluctuation similarity between the data points and the data points to be identified.

[0065] For example, if the distance matrix is ​​an m-row n-column matrix, the path can be determined by traversing from the first element of the distance matrix to the second element of the matrix to determine the path with the minimum sum of similar distances.

[0066] Wherein, the first element is located in the first row and first column of the distance matrix, and the second element is located in the m-th row and n-th column of the distance matrix, which can be understood as traversing from the top left corner to the bottom right corner of the distance matrix; or the first element is located in the m-th row and first column of the distance matrix, and the second element is located in the first row and n-th column of the distance matrix, which can be understood as traversing from the bottom left corner to the top right corner of the distance matrix. It is understood that the starting point of the determined path is relatively fixed.

[0067] In one embodiment, step S120 includes: obtaining a fluctuation similarity sub-score between the fluctuation information of a data point on the baseline data curve and the fluctuation information of a data point to be identified aligned with the data point, based on the correspondence, to obtain a plurality of fluctuation similarity sub-scores; and obtaining the fluctuation similarity score based on the plurality of fluctuation similarity sub-scores.

[0068] Optionally, the fluctuation similarity score can be the average score of multiple waveform similarity sub-scores. The fluctuation similarity score can also be any one of the multiple waveform similarity sub-scores; for example, the fluctuation similarity score can be the maximum or minimum value among the multiple waveform similarity sub-scores.

[0069] In this embodiment, the fluctuation similarity sub-score is calculated for every two points according to the correspondence, resulting in multiple fluctuation similarity sub-scores; then, based on the multiple fluctuation similarity sub-scores, a fluctuation similarity score is obtained. The fluctuation similarity score can take into account the fluctuation factors of more points, making the fluctuation similarity score more able to reflect the fluctuation differences between the two curves, and further improving the accuracy of abnormal data identification.

[0070] As another approach, step S120 includes: dividing the reference data curve into multiple first sub-curves, and dividing the data curve to be identified after being aligned with the reference data curve into multiple second sub-curves; obtaining a fluctuation similarity sub-score between the fluctuation information of each first sub-curve and the fluctuation information of the second sub-curve aligned with the first sub-curve, thereby obtaining multiple fluctuation similarity sub-scores; and obtaining the fluctuation similarity score based on the multiple fluctuation similarity sub-scores.

[0071] Optionally, the fluctuation similarity score can be the average score of multiple waveform similarity sub-scores. The fluctuation similarity score can also be any one of the multiple waveform similarity sub-scores; for example, the fluctuation similarity score can be the maximum or minimum value among the multiple waveform similarity sub-scores.

[0072] In this embodiment, the curve is segmented, and the fluctuation similarity sub-score is obtained according to the segmentation, thereby obtaining the fluctuation similarity score. Compared with the aforementioned point-by-point comparison method, the speed of obtaining the fluctuation similarity score can be improved.

[0073] In one possible implementation, the fluctuation information includes price increase information and upward trend information. The fluctuation similarity sub-score is obtained by: obtaining the upward trend similarity score between the upward trend information of each first sub-curve and the upward trend information of the second sub-curve corresponding to the first sub-curve; obtaining the price increase similarity score between the price increase information of each first sub-curve and the price increase information of the second sub-curve corresponding to the first sub-curve; and obtaining the fluctuation similarity sub-score based on the sum of the upward trend similarity score and the price increase similarity score.

[0074] Optionally, the upward trend similarity score can be obtained based on the first similarity distance between the data points of the first sub-curve. For example, the first similarity distance of any data point on the first sub-curve can be used as the upward trend similarity score. Alternatively, the average of the first similarity distances of all data points on the first sub-curve can be used as the upward trend similarity score.

[0075] Optionally, the increase similarity score can be obtained based on the first similarity distance between the data points of the first sub-curve. For example, the first similarity distance of any data point on the first sub-curve can be used as the increase similarity score. Alternatively, the average of the first similarity distances of all data points on the first sub-curve can be used as the increase similarity score.

[0076] Optionally, after identifying the abnormal data curve, the abnormal data points on the abnormal data curve can be further identified. The method further includes: identifying the data points to be identified corresponding to the fluctuation similarity sub-scores that are greater than the second preset score from the plurality of fluctuation similarity sub-scores as abnormal data points.

[0077] This implementation identifies anomalous data points in the abnormal data curve, facilitating analysis and investigation of the causes of the anomalies. For example, if the abnormal data curve represents the change in sales revenue over time during a promotion, and the sales revenue on the abnormal data curve increases abnormally even when the overall sales revenue is rising, analysis of the abnormal data points could reveal anomalies such as incorrect unit price settings.

[0078] Optionally, the method further includes: obtaining an overall similarity score between the baseline data curve and the data curve to be identified; obtaining a similarity score based on the sum of the overall similarity score and the fluctuation similarity score; and determining the data curve to be identified as an abnormal data curve if the similarity score is greater than a third preset score.

[0079] Please refer to Figure 8 The overall similarity score is obtained as follows: The baseline data curve and the data curve to be identified are analyzed using three models: the first model, the second model, and the third model. For example, for the baseline data curve, the first model fits the data trend of the baseline data curve, which can be represented by the slope. The second model compares the differences between two adjacent data points in the baseline data curve, using the sign of the difference between the previous and subsequent data points to detect the trend; a positive sign indicates an increasing trend, and a negative sign indicates a decreasing trend. Using the third model, for each pair of data points, their difference is calculated, and the sign of the difference is determined. A statistic S is calculated, which is the sum of the signs of all positive differences minus the sum of the signs of all negative differences. Based on the S value and the number of data points, the Z-value of the standard normal distribution is calculated. The significance of the trend is determined based on the Z-value and the significance level. Similarly, for the data curve to be identified, the trend is analyzed using the first, second, and third models respectively. If the trend analysis results using the same model are consistent, the score is 1 point; otherwise, the score is 0 points. It is easy to understand that three scores can be obtained for the analysis results of the three models. The overall similarity score is obtained by using a combined vote. If at least two of the three scores are 1, the overall similarity score is considered to be 1. Alternatively, if at least two of the three scores are 0, the overall similarity score is considered to be 0.

[0080] The above analysis uses different models and a fusion voting method to obtain the final overall similarity score, which improves the accuracy of the overall trend analysis.

[0081] In this embodiment, the overall similarity score can reflect the overall trend, and the fluctuation similarity score can reflect the local trend. The similarity score obtained can reflect both the overall trend and the local trend of the curve. Using the similarity score to identify abnormal data further improves the accuracy of anomaly identification.

[0082] Optionally, there can be multiple data curves to be identified. The abnormal data is identified as follows: Since there are multiple data curves to be identified, there are multiple fluctuation similarity scores. The average of these multiple fluctuation similarity scores is calculated. For each data point in the data curve to be identified, the absolute difference between the waveform similarity sub-score and the aforementioned average is calculated. Then, the average of all absolute differences is calculated. A preset threshold is set based on the average of the absolute differences. This preset threshold is a preset multiple of the average of the absolute differences, such as 2 times, 3 times, etc. When performing anomaly identification, if the difference between the waveform similarity sub-score corresponding to a certain data point to be identified and the average of the absolute differences exceeds the preset threshold, the data point to be identified can be considered an abnormal data point.

[0083] Combining this with the aforementioned method for filtering outlier data points, it can be seen that the second preset score can be the sum of the average absolute difference and the preset threshold.

[0084] The anomaly data identification method disclosed herein can intelligently identify abnormal data curves, reducing losses caused by business anomalies. Aligning the baseline data curve with the data curve to be identified before anomaly identification improves accuracy and reduces false positives, thereby enhancing user experience. Furthermore, compared to existing model-based identification methods, this method eliminates the need for model training and invocation, thus keeping anomaly identification time within 500ms. It can also identify anomaly data by comparing it with other normal data, further improving accuracy. In addition, business testing shows an anomaly data identification precision of up to 82% and a recall rate of approximately 86%.

[0085] Based on the same inventive concept, this disclosure provides an abnormal data identification device. Please refer to [link to relevant documentation]. Figure 9 The abnormal data identification device 200 includes: Alignment module 210 is configured to align the fluctuation trends of a reference data curve and a data curve to be identified, wherein the reference data curve is a normal data curve; The acquisition module 220 is configured to acquire a fluctuation similarity score between fluctuation information of a reference data curve and waveform information of a data curve to be identified after alignment with the reference data curve, wherein the fluctuation information includes increase information and / or increase trend information, and the fluctuation similarity score is inversely correlated with the degree of similarity. The identification module 230 is configured to determine that the data curve to be identified is an abnormal data curve when the fluctuation similarity score is greater than a first preset score.

[0086] In one possible implementation, the reference data curve includes multiple normal data points, the data curve to be identified includes multiple data points to be identified, and the alignment module 210 includes: The matrix determination module is configured to determine the similarity distance between the fluctuation information of each data point on the baseline data curve and the fluctuation information of each data point to be identified on the data curve to be identified, thereby obtaining a distance matrix; The correspondence determination module is configured to determine the correspondence between data points on the reference data curve and data points to be identified on the data curve to be identified, based on the distance matrix.

[0087] In one possible implementation, the correspondence determination module includes: The path determination module is configured to traverse the distance matrix in a preset manner to determine the path with the minimum sum of similar distances; The correspondence construction module is configured to determine the correspondence between data points on the baseline data curve and data points to be identified on the data curve to be identified, based on the path.

[0088] In one possible implementation, the distance matrix is ​​an m-row n-column matrix, and the path determination module is specifically configured to traverse from the first element of the distance matrix to the second element of the matrix to determine the path with the minimum sum of similar distances, wherein the first element is located in the first row and first column of the distance matrix, and the second element is located in the m-th row and n-th column of the distance matrix; or the first element is located in the m-th row and first column of the distance matrix, and the second element is located in the first row and n-th column of the distance matrix.

[0089] In one possible implementation, the acquisition module 220 includes: The sub-score acquisition module is configured to acquire, based on the correspondence, a fluctuation similarity sub-score between the fluctuation information of a data point on the baseline data curve and the fluctuation information of a data point to be identified aligned with that data point, thereby obtaining multiple fluctuation similarity sub-scores. The first score acquisition module is configured to obtain the fluctuation similarity score based on the multiple fluctuation similarity sub-scores.

[0090] In one possible implementation, the abnormal data identification device 200 further includes: The abnormal data point identification module is configured to identify the data points to be identified that correspond to a fluctuation similarity sub-score greater than a second preset score from the plurality of fluctuation similarity sub-scores as abnormal data points.

[0091] In one possible implementation, the acquisition module 220 includes: The segmentation module is configured to divide the reference data curve into multiple first sub-curves and to divide the data curve to be identified after being aligned with the reference data curve into multiple second sub-curves. The calculation module is configured to obtain a fluctuation similarity sub-score between the fluctuation information of each first sub-curve and the fluctuation information of the second sub-curve aligned with the first sub-curve, and obtain multiple fluctuation similarity sub-scores. The second score acquisition module is configured to obtain the fluctuation similarity score based on the multiple fluctuation similarity sub-scores.

[0092] In one possible implementation, the fluctuation information includes price increase information and upward trend information, and the second score acquisition module includes: The upward trend similarity score acquisition module is configured to acquire the upward trend similarity score between the upward trend information of each first sub-curve and the upward trend information of the second sub-curve corresponding to the first sub-curve; The similarity score acquisition module is configured to acquire the similarity score between the increase information of each first sub-curve and the increase information of the second sub-curve corresponding to the first sub-curve; The volatility similarity sub-score calculation module is configured to obtain the volatility similarity sub-score based on the sum of the upward trend similarity score and the increase similarity score.

[0093] In one possible implementation, the abnormal data identification device 200 further includes: The overall similarity score acquisition module is configured to acquire the overall similarity score between the benchmark data curve and the data curve to be identified; The similarity score acquisition module is configured to obtain a similarity score based on the sum of the overall similarity score and the fluctuation similarity score; The anomaly detection module is configured to determine that the data curve to be identified is an anomalous data curve if the similarity score is greater than a third preset score.

[0094] Regarding the abnormal data identification device 200 in the above embodiments, the specific methods by which each module performs its operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0095] This disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the steps of the abnormal data identification method provided in this disclosure.

[0096] Figure 10This is a block diagram illustrating an electronic device for an anomaly data identification method according to an exemplary embodiment. For example, the electronic device 800 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0097] Please refer to Figure 10 The electronic device 800 may include one or more of the following components: processing component 802, memory 804, power supply component 806, multimedia component 808, audio component 810, input / output interface 812, sensor component 814, and communication component 816.

[0098] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0099] Memory 804 is configured to store various types of data to support the operation of electronic device 800. Examples of such data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0100] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0101] Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0102] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0103] Input / output interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0104] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 can detect the on / off state of electronic device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or a component of electronic device 800, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0105] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0106] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0107] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0108] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the above-described abnormal data identification method when executed by the programmable device.

[0109] Those skilled in the art will also understand that the various illustrative logical blocks and steps listed in the embodiments of this application can be implemented by electronic hardware, computer software, or a combination of both. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the described functionality using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of this application.

[0110] Furthermore, the term “exemplary” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as advantageous compared to other aspects or designs. Rather, the use of the term “exemplary” is intended to present the concept in a concrete manner. As used herein, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or clear from the context, “X applies A or B” is intended to mean any of the natural inclusive arrangements. That is, “X applies A or B” satisfies any of the foregoing instances if X applies A; X applies B; or both X applies A and B. Additionally, unless otherwise specified or clear from the context to refer to the singular form, the articles “a” and “an” as used in this application and the appended claims are generally understood to mean “one or more.”

[0111] Similarly, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding this specification and the accompanying drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, the terminology used to describe such components is intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if structurally not equivalent to the disclosed structure. Furthermore, although specific features of this disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations, as may be desired and advantageous to any given or particular application. Moreover, with regard to the terms “comprising,” “owning,” “having,” “having,” or variations thereof as used in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term “including.”

[0112] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

[0113] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for identifying abnormal data, characterized in that, The method includes: Align the fluctuation trends of the baseline data curve and the data curve to be identified, wherein the baseline data curve is a normal data curve; A fluctuation similarity score is obtained between the fluctuation information of the baseline data curve and the waveform information of the data curve to be identified after being aligned with the baseline data curve, wherein the fluctuation information includes the increase information and / or the upward trend information, and the fluctuation similarity score is inversely correlated with the degree of similarity; If the fluctuation similarity score is greater than the first preset score, the data curve to be identified is determined to be an abnormal data curve.

2. The abnormal data identification method according to claim 1, characterized in that, The baseline data curve includes multiple normal data points, and the data curve to be identified includes multiple data points to be identified. The baseline data curve and the data curve to be identified are aligned in the following manner: Determine the similarity distance between the fluctuation information of each data point on the baseline data curve and the fluctuation information of each data point on the data curve to be identified, and obtain a distance matrix; Based on the distance matrix, the correspondence between the data points on the reference data curve and the data points to be identified on the data curve to be identified is determined.

3. The abnormal data identification method according to claim 2, characterized in that, The step of determining the correspondence between data points on the reference data curve and data points on the data curve to be identified based on the distance matrix includes: The distance matrix is ​​traversed according to a preset method to determine the path with the minimum sum of similar distances; Based on the path, determine the correspondence between the data points on the baseline data curve and the data points to be identified on the data curve to be identified.

4. The abnormal data identification method according to claim 3, characterized in that, The distance matrix is ​​an m x n matrix. The step of traversing the distance matrix according to a preset method to determine the path with the minimum sum of similar distances includes: Starting from the first element of the distance matrix, traverse to the second element of the matrix to determine the path with the minimum sum of similar distances, wherein the first element is located in the first row and first column of the distance matrix, and the second element is located in the m-th row and n-th column of the distance matrix; or the first element is located in the m-th row and first column of the distance matrix, and the second element is located in the first row and n-th column of the distance matrix.

5. The abnormal data identification method according to claim 2, characterized in that, The fluctuation similarity score between the fluctuation information of the benchmark data curve and the fluctuation information of the data curve to be identified after alignment with the benchmark data curve includes: Based on the correspondence, the fluctuation similarity sub-score between the fluctuation information of the data point on the benchmark data curve and the fluctuation information of the data point to be identified aligned with the data point is obtained, resulting in multiple fluctuation similarity sub-scores. The fluctuation similarity score is obtained based on the multiple fluctuation similarity sub-scores.

6. The abnormal data identification method according to claim 5, characterized in that, Also includes: From the multiple fluctuation similarity sub-scores, the data points to be identified corresponding to fluctuation similarity sub-scores greater than the second preset score are determined as abnormal data points.

7. The abnormal data identification method according to claim 1, characterized in that, The fluctuation similarity score between the fluctuation information of the benchmark data curve and the waveform information of the data curve to be identified after alignment with the benchmark data curve includes: The baseline data curve is divided into multiple first sub-curves, and the data curve to be identified, after being aligned with the baseline data curve, is divided into multiple second sub-curves. The fluctuation similarity sub-score is obtained between the fluctuation information of the first sub-curve and the fluctuation information of the second sub-curve aligned with the first sub-curve, resulting in multiple fluctuation similarity sub-scores. The fluctuation similarity score is obtained based on the multiple fluctuation similarity sub-scores.

8. The abnormal data identification method according to claim 7, characterized in that, The fluctuation information includes price increase information and upward trend information, and the fluctuation similarity sub-score is obtained through the following method: Obtain the similarity score of the upward trend between the upward trend information of each first sub-curve and the upward trend information of the second sub-curve corresponding to the first sub-curve; Obtain the similarity score of the increase information between the increase information of each first sub-curve and the increase information of the corresponding second sub-curve; The volatility similarity sub-score is obtained by summing the upward trend similarity score and the increase similarity score.

9. The abnormal data identification method according to any one of claims 1 to 8, characterized in that, Also includes: Obtain the overall similarity score between the baseline data curve and the data curve to be identified; A similarity score is obtained by summing the overall similarity score and the fluctuation similarity score; If the similarity score is greater than a third preset score, the data curve to be identified is determined to be an abnormal data curve.

10. An abnormal data identification device, characterized in that, include: The alignment module is configured to align the fluctuation trends of a reference data curve and a data curve to be identified, wherein the reference data curve is a normal data curve; The acquisition module is configured to acquire a fluctuation similarity score between fluctuation information of a baseline data curve and waveform information of a data curve to be identified after alignment with the baseline data curve, wherein the fluctuation information includes increase information and / or upward trend information, and the fluctuation similarity score is inversely correlated with the degree of similarity; The identification module is configured to determine that the data curve to be identified is an abnormal data curve when the fluctuation similarity score is greater than a first preset score.

11. The abnormal data identification device according to claim 10, characterized in that, The alignment module includes: The matrix determination module is configured to determine the similarity distance between the fluctuation information of each data point on the baseline data curve and the fluctuation information of each data point to be identified on the data curve to be identified, thereby obtaining a distance matrix; The correspondence determination module is configured to determine the correspondence between data points on the reference data curve and data points to be identified on the data curve to be identified, based on the distance matrix.

12. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the steps of the method according to any one of claims 1 to 9 when executing the instruction.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method described in any one of claims 1 to 9.

14. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

Images