A data outlier detection method based on box plot index

Abstract

The application discloses a data outlier detection method based on a box plot index, and is characterized in that the method comprises the following steps: S1, establishing a box plot index calculation and data outlier detection system; S2, collecting orders generated by users in real time through a data collection module, and delivering order data to a downstream box plot index calculation module and the like. The application can calculate the distribution index of data in real time with high performance by separately processing different indexes of the box plot, avoids high data delay caused by data sorting, and thus can more quickly show the complete distribution of data and detect outliers, so that a merchant can quickly master the data distribution and find abnormal values in data in advance, the merchant can find abnormal orders in time and quickly investigate the abnormal reasons, and the application can help the merchant reduce more losses caused by abnormal orders.

Description

Technical Field

[0001] This invention belongs to the field of software technology, specifically referring to a method for detecting outliers in data based on box plot indicators. Background Technology

[0002] During promotional periods for e-commerce platforms and retailers, merchants typically display order-related statistical metrics, such as total orders and averages, on digital dashboards. Beyond providing simple data indicators, they also offer statistical graphs for more in-depth analysis. Box plots are a widely used standard method that uses five numbers to describe the distribution of data, allowing merchants to clearly understand the median, central tendency, and statistical extremes of current orders. Box plots can also be used to detect outliers, assisting merchants in investigating the causes of risky or abnormal orders.

[0003] However, existing methods for detecting outliers using box plots cannot detect outliers in a timely manner when importing large amounts of data. This prevents merchants from promptly identifying abnormal orders, resulting in significant economic losses for them. Summary of the Invention

[0004] The purpose of this invention is to overcome the problem that existing methods for detecting outliers using box plots cannot detect outliers in a timely manner when the imported data is too large. This invention provides a method for detecting outliers based on box plot indicators that can quickly calculate relevant box plot indicators and perform rapid outlier detection on the input data. This allows merchants to quickly understand the distribution of order data and identify abnormal orders in advance, helping them to quickly investigate abnormal situations such as human-induced pricing errors and malicious purchases.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] A method for detecting outliers in data based on box plot indicators includes the following steps:

[0007] S1: Establish a system for calculating box plot indicators and detecting outliers in the data;

[0008] The box plot index calculation system includes a data collection module and a box plot index calculation module; the data outlier detection module system includes an outlier detection module, an outlier message sending module, and a data visualization module.

[0009] S2: The data collection module collects user-generated orders in real time and sends the order data to the downstream box plot indicator calculation module;

[0010] S3: The box plot indicator calculation module performs grouped calculations on the received order data, and calculates the latest median, upper quartile, and lower quartile indicators in the entire data stream using a storage method based on maxHeap and minHeap. After excluding outliers by calculating based on the TreeSet dataset, the maximum and minimum extreme values ​​in the entire data stream are obtained, resulting in five indicators of the box plot. The five indicators of the box plot and the order data are then sent to the outlier detection module.

[0011] S4: The outlier detection module summarizes the five indicators of the box plot and determines whether the order data is an outlier based on the summarized box plot indicators; if yes, it is tagged and sent to the outlier message sending module; if no, the order data is displayed according to the distribution of the box plot indicators through the data visualization module.

[0012] S5: The outlier message sending module sends outlier information to the merchant and also sends it to the data visualization module for display. The outlier detection is now complete.

[0013] In a preferred embodiment of the present invention, in step S1, the box plot index calculation module includes:

[0014] The median calculation module is used to calculate the median value in order data;

[0015] The upper quartile calculation module is used to calculate the upper quartile index in order data;

[0016] The lower quartile calculation module is used to calculate the lower quartile index in order data;

[0017] The extreme value calculation module is used to calculate the maximum and minimum extreme values ​​in the order data. The extreme value calculation module mainly consists of a dataset based on TreeSet.

[0018] Furthermore, the median calculation module, the upper quartile calculation module, and the lower quartile calculation module are each equipped with a heap adjustment module;

[0019] The median calculation module is configured with a parameter n, where n = 1, and the parameter n is the ratio of the maxHeap (maximum heap) parameter to the minHeap parameter; the upper quartile calculation module is configured with a parameter that is one-third of the median calculation module parameter n, and the lower quartile calculation module is configured with a parameter that is three times the median calculation module parameter n.

[0020] The heap adjustment module is used to adjust the data size of maxHeap and minHeap to ensure that maxHeap and minHeap meet the quantity rules, which are as follows:

[0021] The number of elements in n×minHeap is less than or equal to the number of elements in maxHeap, which is less than or equal to the number of elements in n×minHeap plus 1.

[0022] Furthermore, in step S3, the calculation methods for the median, upper quartile, and lower quartile are the same in the storage method based on maxHeap plus minHeap. Specifically, the steps for calculating the median based on the storage method of maxHeap plus minHeap are as follows:

[0023] a. The box plot indicator calculation module performs stacking processing on the input order data to obtain maxHeap and minHeap;

[0024] b. Determine whether the data of maxHeap and minHeap obtained by the heap adjustment module meet the quantity rules; if yes, proceed to step c; if no, the heap adjustment module adjusts the data size of maxHeap and minHeap, and after the adjustment is completed, proceed to step c.

[0025] c. The median calculation module performs slice determination on maxHeap and minHeap that meet the quantity rules, and determines whether the length of maxHeap is greater than n×minHeap length; if yes, the top element of maxHeap is extracted, and this element is the median index; if no, the median index is obtained by calculating the formula: median index = (top element of maxHeap + top element of minHeap) / 2.

[0026] In a preferred embodiment of the present invention, the steps for processing order data in step a are as follows:

[0027] a1. The line chart indicator calculation module identifies whether maxHeap in the input order data is empty; if no, proceed to step a2; if yes, directly insert the data into maxHeap.

[0028] a2. Determine whether the input data is less than or equal to the top element of maxHeap; if yes, insert the data into maxHeap; if no, insert the data into minHeap.

[0029] In a preferred embodiment of the present invention, step b involves the heap adjustment module adjusting the data sizes of maxHeap and minHeap as follows:

[0030] b1. Determine if the length of maxHeap is less than the length of minHeap. If yes, move the top element of minHeap into maxHeap to obtain maxHeap and minHeap that meet the quantity rules, and the adjustment ends. If no, proceed to step b2.

[0031] b2. Determine if the length of maxHeap is greater than n × minHeap length + 1; if yes, move the top element of maxHeap into minHeap to obtain maxHeap and minHeap that satisfy the quantity rule, and the adjustment ends; if no, the adjustment ends.

[0032] In a preferred embodiment of the present invention, step S3, which involves calculating and excluding outliers using the TreeSet dataset to determine the maximum and minimum extreme values ​​in the entire data stream, is as follows:

[0033] (1) Insert each data point in the order data directly into the TreeSet dataset of the extreme value calculation module;

[0034] (2) Calculate the interquartile range (IQR) by using the upper quartile and lower quartile indices obtained by the storage method based on maxHeap plus minHeap, where the interquartile range (IQR) = lower quartile index - upper quartile index;

[0035] (3) By calling the floor method of the dataset and using the maximum value = lower four-digit index + 1.5 × interquartile range (IQR), the maximum value is directly obtained from the order data points in the TreeSet dataset of the insertion extreme value calculation module;

[0036] (4) By calling the ceiling method of the dataset, and by using the minimum extreme value = upper quartile index - 1.5 × interquartile range (IQR), the minimum value is directly obtained from the order data points in the TreeSet dataset of the inserted extreme value calculation module.

[0037] Furthermore, in step S4, the method for determining whether order data is an outlier based on the summarized box plot indicators is as follows: the outlier detection module determines whether a certain data point in the order data is greater than the lower four-digit indicator + 1.5 × interquartile range (IQR) or less than the upper quartile indicator - 1.5 × interquartile range (IQR) based on the lower four-digit indicator, upper quartile indicator, and interquartile range (IQR) calculated by the box plot indicator calculation module.

[0038] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0039] This invention primarily utilizes a method of processing different indicators in box plots separately, enabling real-time calculation of data distribution indicators with high performance. This avoids the high data latency caused by data sorting, thereby allowing for a faster display of the complete data distribution and outlier detection. This facilitates merchants in quickly understanding data distribution and identifying outliers in advance. Merchants can promptly detect abnormal orders and quickly investigate the causes of anomalies, helping them reduce further losses caused by abnormal orders. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the overall structure of the present invention.

[0041] Figure 2 This is a flowchart illustrating the overall implementation of the present invention.

[0042] Figure 3 This is a flowchart of the calculation process for the box plot index calculation module of the present invention. Detailed Implementation

[0043] The present invention will be further described in detail below with reference to embodiments, but the implementation of the present invention is not limited thereto.

[0044] Example

[0045] like Figures 1-3 As shown, the objective of this invention is achieved through the following technical solution: A method for detecting outliers in data based on box plot indicators includes the following steps:

[0046] S1: Establish a box plot index calculation and outlier detection system. Specifically, the box plot index calculation system includes a data collection module and a box plot index calculation module; the outlier detection module system includes an outlier detection module, an outlier message sending module, and a data visualization module.

[0047] The box plot index calculation module includes:

[0048] The median calculation module is used to calculate the median value in the order data.

[0049] The upper quartile calculation module is used to calculate the upper quartile index in order data.

[0050] The lower quartile calculation module is used to calculate the lower quartile index in order data.

[0051] The extreme value calculation module is used to calculate the maximum and minimum extreme values ​​in the order data. The extreme value calculation module mainly consists of a dataset based on TreeSet.

[0052] Simultaneously, the median calculation module, upper quartile calculation module, and lower quartile calculation module are each equipped with a heap adjustment module. A parameter n is configured for the median calculation module, where n = 1, and parameter n is the ratio of the maxHeap (maximum heap) parameter to the minHeap parameter. The parameter configured for the upper quartile calculation module is one-third of the parameter n of the median calculation module, and the parameter configured for the lower quartile calculation module is three times the parameter n of the median calculation module. The heap adjustment module is used to adjust the data size of maxHeap and minHeap to ensure that maxHeap and minHeap meet the following quantity rules:

[0053] The number of elements in n×minHeap is less than or equal to the number of elements in maxHeap, which is less than or equal to the number of elements in n×minHeap plus 1.

[0054] S2: The data collection module collects user-generated orders in real time and sends the order data to the downstream box plot indicator calculation module.

[0055] S3: The box plot indicator calculation module performs grouped calculations on the received order data, and calculates the latest median, upper quartile, and lower quartile indicators in the entire data stream using a storage method based on maxHeap and minHeap. After excluding outliers by calculating based on the TreeSet dataset, the maximum and minimum extreme values ​​in the entire data stream are obtained, resulting in five indicators for the box plot. The five indicators of the box plot and the order data are then sent to the outlier detection module.

[0056] In practical implementation, the calculation methods for the median, upper quartile, and lower quartile indices are the same when using the maxHeap plus minHeap storage method. For example, Figure 3 As shown, the steps for calculating the median index based on the storage method of maxHeap plus minHeap are as follows:

[0057] a) The box plot indicator calculation module performs heap partitioning on the input order data to obtain maxHeap and minHeap. Specifically, the heap partitioning process in step a) is as follows:

[0058] a1. The line chart indicator calculation module identifies whether maxHeap in the input order data is empty; if no, proceed to step a2; if yes, directly insert the data into maxHeap.

[0059] a2. Determine whether the input data is less than or equal to the top element of maxHeap; if yes, insert the data into maxHeap; if no, insert the data into minHeap.

[0060] b. The heap adjustment module determines whether the obtained maxHeap and minHeap data meet the quantity rules; if yes, proceed to step c; if no, the heap adjustment module adjusts the data sizes of maxHeap and minHeap, and after the adjustment is completed, proceed to step c. Specifically, the heap adjustment module adjusts the data sizes of maxHeap and minHeap as follows:

[0061] b1. Determine if the length of maxHeap is less than the length of minHeap. If yes, move the top element of minHeap into maxHeap to obtain maxHeap and minHeap that meet the quantity rules, and the adjustment ends. If no, proceed to step b2.

[0062] b2. Determine if the length of maxHeap is greater than n × minHeap length + 1; if yes, move the top element of maxHeap into minHeap to obtain maxHeap and minHeap that satisfy the quantity rule, and the adjustment ends; if no, the adjustment ends.

[0063] c. The median calculation module performs slice determination on maxHeap and minHeap that meet the quantity rules, and determines whether the length of maxHeap is greater than n×minHeap length; if yes, the top element of maxHeap is extracted, and this element is the median index; if no, the median index is obtained by calculating the formula: median index = (top element of maxHeap + top element of minHeap) / 2.

[0064] Additionally, the steps to calculate and exclude outliers using the TreeSet dataset to determine the maximum and minimum extreme values ​​in the entire data stream are as follows:

[0065] (1) Insert each data point in the order data directly into the TreeSet dataset of the extreme value calculation module.

[0066] (2) Calculate the interquartile range (IQR) by using the upper quartile and lower quartile indices obtained by the storage method based on maxHeap plus minHeap, where IQR = lower quartile index - upper quartile index.

[0067] (3) By calling the floor method of the dataset, and by using the maximum value = lower four-digit index + 1.5 × interquartile range (IQR), the maximum value is directly obtained from the order data points in the TreeSet dataset of the insertion extreme value calculation module.

[0068] (4) By calling the ceiling method of the dataset, and by using the minimum extreme value = upper quartile index - 1.5 × interquartile range (IQR), the minimum value is directly obtained from the order data points in the TreeSet dataset of the inserted extreme value calculation module.

[0069] S4: The outlier detection module summarizes the five indicators of the box plot and determines whether the order data is an outlier based on the summarized box plot indicators; if yes, it is tagged and sent to the outlier message sending module; if no, the order data is displayed according to the distribution of the box plot indicators through the data visualization module.

[0070] The method for determining whether order data is an outlier based on the summarized box plot indicators is as follows: The outlier detection module determines whether a data point in the order data is greater than the lower four-digit indicator + 1.5 × interquartile range (IQR) or less than the upper quartile indicator - 1.5 × interquartile range (IQR) based on the lower four-digit indicator, upper quartile indicator, and interquartile range (IQR) calculated by the box plot indicator calculation module.

[0071] S5: The outlier message sending module sends outlier information to the merchant and also sends it to the data visualization module for display. The outlier detection is now complete.

[0072] This invention primarily utilizes a method of processing different indicators in box plots separately, enabling real-time calculation of data distribution indicators with high performance. This avoids the high data latency caused by data sorting, thereby allowing for a faster display of the complete data distribution and outlier detection. This facilitates merchants in quickly understanding data distribution and identifying outliers in advance. Merchants can promptly detect abnormal orders and quickly investigate the causes of anomalies, helping them reduce further losses caused by abnormal orders.

[0073] As described above, the present invention can be well implemented.

Claims

1. A method for detecting outliers in data based on box plot indicators, characterized in that, Includes the following steps: S1: Establish a box plot index calculation system and a data outlier detection module system; The box plot index calculation system includes a data collection module and a box plot index calculation module; the data outlier detection module system includes an outlier detection module, an outlier message sending module, and a data visualization module. S2: The data collection module collects user-generated orders in real time and sends the order data to the downstream box plot indicator calculation module; S3: The box plot indicator calculation module performs grouped calculations on the received order data, and calculates the latest median, upper quartile, and lower quartile indicators in the entire data stream using a storage method based on maxHeap and minHeap. After excluding outliers by calculating based on the TreeSet dataset, the maximum and minimum extreme values ​​in the entire data stream are obtained, resulting in five indicators of the box plot. The five indicators of the box plot and the order data are then sent to the outlier detection module. S4: The outlier detection module summarizes the five indicators of the box plot and determines whether the order data is an outlier based on the summarized box plot indicators; if yes, it is tagged and sent to the outlier message sending module; if no, the order data is displayed according to the distribution of the box plot indicators through the data visualization module. S5: The outlier message sending module sends outlier information to the merchant and also sends it to the data visualization module for display. The outlier detection is now complete.

2. The outlier detection method based on box plot indicators according to claim 1, characterized in that, In step S1, the box plot indicator calculation module includes: The median calculation module is used to calculate the median value in order data; The upper quartile calculation module is used to calculate the upper quartile index in order data; The lower quartile calculation module is used to calculate the lower quartile index in order data; The extreme value calculation module is used to calculate the maximum and minimum extreme values ​​in the order data. The extreme value calculation module mainly consists of a dataset based on TreeSet. The median calculation module, the upper quartile calculation module, and the lower quartile calculation module are each equipped with a heap adjustment module; The median calculation module is configured with a parameter n, where n=1, and the parameter n is the ratio of the maxHeap to the minHeap parameters; the upper quartile calculation module is configured with a parameter that is one-third of the median calculation module parameter n, and the lower quartile calculation module is configured with a parameter that is three times the median calculation module parameter n. The heap adjustment module is used to adjust the data size of maxHeap and minHeap to ensure that maxHeap and minHeap meet the quantity rules, which are as follows: The number of elements in n×minHeap is less than or equal to the number of elements in maxHeap, which is less than or equal to the number of elements in n×minHeap plus 1.

3. The outlier detection method based on box plot indicators according to claim 2, characterized in that, In step S3, the calculation methods for the median, upper quartile, and lower quartile indices based on the storage method of maxHeap plus minHeap are the same. Specifically, the steps for calculating the median index based on the storage method of maxHeap plus minHeap are as follows: a. The box plot indicator calculation module performs stacking processing on the input order data to obtain maxHeap and minHeap; b. Determine whether the data of maxHeap and minHeap obtained by the heap adjustment module meet the quantity rules; if yes, proceed to step c; if no, the heap adjustment module adjusts the data size of maxHeap and minHeap, and after the adjustment is completed, proceed to step c. c. The median calculation module performs slice determination on maxHeap and minHeap that meet the quantity rules, and determines whether the length of maxHeap is greater than n × the length of minHeap; if yes, the top element of maxHeap is extracted, and this element is the median index; if no, the median index is obtained by calculating the formula: median index = (top element of maxHeap + top element of minHeap) / 2.

4. The outlier detection method based on box plot indicators according to claim 3, characterized in that, In step a, the order data is processed by piling up as follows: a1. The line chart indicator calculation module identifies whether maxHeap in the input order data is empty; if no, proceed to step a2; if yes, directly insert the data into maxHeap. a2. Determine whether the input data is less than or equal to the top element of maxHeap; Yes, insert the data into maxHeap; no, insert the data into minHeap.

5. The outlier detection method based on box plot indicators according to claim 4, characterized in that, In step b, the heap adjustment module adjusts the data size of maxHeap and minHeap as follows: b1. Determine if the length of maxHeap is less than the length of minHeap; if yes, move the top element of minHeap into maxHeap to obtain maxHeap and minHeap that satisfy the quantity rule, and the adjustment ends. No, proceed to step b2; b2. Determine if the length of maxHeap is greater than n × minHeap length + 1; Yes, move the top element of maxHeap into minHeap to obtain maxHeap and minHeap that satisfy the quantity rules, and the adjustment is complete; No, adjustment complete.

6. The outlier detection method based on box plot indicators according to claim 5, characterized in that, In step S3, the steps for calculating and excluding outliers using the TreeSet dataset to obtain the maximum and minimum extreme values ​​in the entire data stream are as follows: (1) Insert each data point in the order data directly into the TreeSet dataset of the extreme value calculation module; (2) Calculate the interquartile range by using the upper quartile and lower quartile indices obtained by the storage method based on maxHeap plus minHeap, where the interquartile range = lower quartile index - upper quartile index. (3) By calling the floor method of the dataset and obtaining the maximum value directly from the order data points in the TreeSet dataset of the extreme value calculation module by using the maximum value = the lower four-digit index + 1.5 × interquartile range; (4) By calling the ceiling method of the dataset, and by using the minimum extreme value = upper quartile index - 1.5 × interquartile range, the minimum value is directly obtained from the order data points in the TreeSet dataset of the insertion extreme value calculation module.

7. The outlier detection method based on box plot indicators according to claim 6, characterized in that, In step S4, the method for determining whether order data is an outlier based on the summarized box plot indicators is as follows: the outlier detection module determines whether a certain data point in the order data is greater than the lower four-digit indicator + 1.5 × interquartile range or less than the upper quartile indicator - 1.5 × interquartile range based on the lower four-digit indicator, upper quartile indicator, and interquartile range calculated by the box plot indicator calculation module.

Images