A data set aided checking method and device based on improved box plot
By dynamically calculating the quantile coefficient k and adaptively adjusting the box plot algorithm, the problem of misjudgment in traditional box plots under complex data distributions is solved, and more efficient anomaly detection is achieved.
Patent Information
- Application Number
- CN202610725405.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-05-25
- Publication Date
- 2026-08-25
AI Technical Summary
Traditional box plot algorithms have a high false positive rate when dealing with complex data distributions and are difficult to adapt to multimodal and asymmetric distributions, resulting in insufficient accuracy in anomaly detection.
By dynamically calculating the quantile coefficient k, a recommendation table of optimal k values for features is generated based on the discrete coefficients of the dataset. The k value is adaptively adjusted to balance sensitivity and false alarm rate, thereby improving verification accuracy.
It improves the accuracy and efficiency of dataset verification and can identify anomalous data in complex distributions.
Smart Images
Figure CN122634434A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of data quality control technology, and in particular relates to a dataset auxiliary verification method and apparatus based on improved box plots. Background Technology
[0002] In the field of data anomaly detection, traditional box plot algorithms are widely used in data distribution visualization due to their intuitiveness and computational efficiency. However, their core relies on a static threshold mechanism with a fixed interquartile range (IQR), which implicitly assumes that the data follows a normal distribution.
[0003] Specifically, box plots determine whether data is outlier based on quantiles, particularly the upper and lower quartiles. The top and bottom edges of the box correspond to the upper quartiles, respectively. (i.e., the data at the 3 / 4 position after the dataset is sorted by size) and the lower quartile. (That is, the data in the 1 / 4 position after the dataset is sorted by size). The formulas for calculating the upper and lower whiskers are as follows: ) Where k is the set quantile coefficient, the upper quartile. lower quartiles Upper quartile position lower quartile position , This indicates the sample size. If the upper quartile position... and lower quartile position If the result is an integer, take the corresponding value directly; otherwise, take the average of the nearest values above and below.
[0004] However, traditional boxplot algorithms not only oversimplify the representation of data distribution, failing to reflect complex structures such as skewness and multimodality, leading to information distortion, but also rely on fixed thresholds (such as 1.5×IQR) for outlier detection, resulting in poor adaptability to different distributions and a high risk of misjudgment. These shortcomings stem from their inherent reliance on static statistics and lack of adaptive mechanisms. Furthermore, since data generated in real-world scenarios often exhibits multimodal distributions, asymmetric skewness, or time-varying drift characteristics, traditional boxplot algorithms are prone to increased false alarm rates due to rigid thresholds. Summary of the Invention
[0005] This invention aims to address the shortcomings of traditional boxplots, such as insufficient distribution representation and poor adaptability in anomaly detection. It proposes a dataset-assisted verification method and device based on improved boxplots, which can improve the accuracy, intelligence, and analysis efficiency of dataset verification.
[0006] To achieve the above objectives, the technical solution of the present invention includes the following:
[0007] A dataset-assisted verification method based on improved boxplots, the method comprising: Generate a dataset to be verified; Generate quantile coefficients based on the coefficients of dispersion of the dataset to be checked; Based on the quantile coefficient, box plot verification is performed on the dataset to be verified to obtain the verification result of the dataset to be verified.
[0008] Further, a dataset to be verified is generated, including: Extract the quantitative attribute data from the original dataset; The quantitative attribute data is sorted in ascending order to obtain the dataset to be verified.
[0009] Furthermore, based on the discrete coefficients of the dataset to be verified, quantile coefficients are generated, including: Calculate the average value of the dataset to be verified; Calculate the standard deviation of the dataset to be verified; Calculate the coefficient of variation of the dataset to be checked based on the mean and standard deviation of the data. Based on this coefficient of variation, the quantile coefficients are obtained.
[0010] Furthermore, based on the quantile coefficient, boxplot verification is performed on the dataset to be verified to obtain the verification results for the dataset to be verified, including: Calculate the upper and lower quartile positions based on the amount of data in the dataset to be verified; Calculate the upper quartile and lower quartile based on their respective positions. Calculate the upper and lower whiskers based on the quantile coefficients, the upper quartile, and the lower quartile; Based on the upper and lower requirements, abnormal data is investigated to obtain the verification results of the dataset to be verified.
[0011] A dataset-assisted verification device based on improved boxplots, the device comprising: The dataset generation module is used to generate the dataset to be verified. The quantile coefficient generation module is used to generate quantile coefficients based on the discrete coefficients of the dataset to be checked. The verification result generation module is used to perform box plot verification on the dataset to be verified based on the quantile coefficient, and obtain the verification result of the dataset to be verified.
[0012] A computer device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the dataset-assisted verification method based on the improved boxplot as described above.
[0013] A computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the dataset-assisted verification method based on the improved boxplot as described above.
[0014] A computer program product, characterized in that, when the computer program product is run on a computer device, the computer device executes the dataset-assisted verification method based on the improved boxplot as described above.
[0015] Compared with existing technologies, this invention dynamically calculates the quantile coefficient k based on the discrete coefficient of the dataset, establishes a recommended table of optimal k values for each attribute, and dynamically adjusts the k value by the discrete coefficient, which can adapt to the high heterogeneity of the data, balance sensitivity and false alarm rate, and improve the verification accuracy. Attached Figure Description
[0016] Figure 1 This is a flowchart of a dataset-assisted verification method based on improved boxplots.
[0017] Figure 2 This is a sequence diagram of preparation time for one embodiment.
[0018] Figure 3 This is a block diagram of a dataset-assisted verification device based on an improved boxplot.
[0019] Figure 4 A block diagram of a computer device. Detailed Implementation
[0020] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, and to make the objectives, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.
[0021] The dataset-assisted verification method of the present invention, such as Figure 1 As shown, it includes the following steps.
[0022] Step 1: Generate the dataset to be verified.
[0023] This invention first filters out the categorical attribute data in the original dataset and extracts the quantitative attribute data suitable for box plot analysis in the original dataset, in preparation for box plot analysis. Then, the quantitative attribute data is sorted in ascending order to generate the dataset to be verified.
[0024] In one embodiment, when the original dataset is an experimental operation dataset, it mainly includes metrics such as experiment number, experiment name, preparation time, operation time, total time, experimenters, experiment status, and experiment type. Therefore, this embodiment first filters out the type attribute data in the original dataset, such as experiment number, experiment name, experimenters, experiment status, and experiment type, and extracts the quantitative attribute data, such as preparation time, operation time, and total time. Then, for a specific quantitative attribute data (such as preparation time), this invention sorts the data in ascending order to obtain the dataset to be verified. Figure 2 A timeline of preparation times for this embodiment.
[0025] Step 2: Generate quantile coefficients based on the discrete coefficients of the dataset to be checked.
[0026] Unlike traditional box plot algorithms where quantile coefficients are obtained by setting them, this invention dynamically calculates quantile coefficients (k) based on the discrete coefficients of the dataset, establishes a recommended table of optimal k values for each attribute, and dynamically adjusts the k value by the discrete coefficients. This can adapt to the high heterogeneity of different types of data, balance sensitivity and false alarm rate, and improve verification accuracy.
[0027] In one embodiment, the process of generating quantile coefficients includes the following sub-steps.
[0028] Step 2.1: Calculate the average value of the dataset to be verified. ,in, The amount of data in the dataset to be verified. For index Quantitative attribute data.
[0029] Step 2.2: Calculate the standard deviation of the dataset to be verified. .
[0030] Step 2.3: Calculate the coefficients of discrepancy of the dataset to be verified. .
[0031] Step 2.4: Calculate the quantile coefficients ,in, Indicates weight, Indicates bias.
[0032] Preferably, it can be set ,and The quantile coefficients are limited to a maximum value of 3 and a minimum value of 0.5. That is, the calculated quantile coefficients... In this case, take the quantile coefficient. .
[0033] Step 3: Perform box plot verification on the dataset to be verified based on the quantile coefficient to obtain the verification result of the dataset to be verified.
[0034] In the box plot algorithm, in addition to obtaining the quantile coefficients... In addition, the upper quartiles also need to be calculated. and lower quartile Only in this way can the upper and lower fibers used for verification data be obtained.
[0035] Please refer to Figure 2 The dataset verification process of the present invention, as shown in the embodiment, includes the following sub-steps.
[0036] Step 3.1: Calculate the upper quartile position based on the data size of the dataset to be verified. and lower quartile position .
[0037] Step 3.2: Based on the upper quartile positions respectively and lower quartile position Calculate the upper quartiles and lower quartile .
[0038] Because of the upper quartile position =6.75 and the lower quartile position =2.25 is not an integer, therefore it is an upper quartile. and lower quartile Take the average of the adjacent values above and below.
[0039] In this embodiment, the upper quartile Take the average of the 6th and 7th data points, that is... = = 22.5; and the lower quartile Take the average of the second and third data points, that is... = = 11.5.
[0040] Step 3.3: Based on quantile coefficients Upper quartiles and lower quartile Calculate the upper and lower beards.
[0041] Upper mustache = = 22.5 + 3*(22.5 - 11.5) = 55.5; Lower beard = =11.5–3*(22.5-11.5) = -28.5 (invalid, take the lower limit 0).
[0042] Step 3.4: Based on the upper and lower limits, perform abnormal data inspection to obtain the verification results of the dataset to be verified.
[0043] In this embodiment, it can be determined that the preparation time of 1000 for a certain experimental operation 4 is abnormal data.
[0044] In summary, this invention combines commonly used mathematical statistics methods for detecting outlier data with an independently improved algorithm. It dynamically calculates quantile coefficients based on data characteristics to identify outlier data in the dataset, providing a basis for outlier data identification while exploring data patterns.
[0045] Based on the same concept, the present invention also provides a dataset-assisted verification device based on an improved boxplot, such as... Figure 3 As shown, the device includes: The dataset generation module is used to generate the dataset to be verified. The quantile coefficient generation module is used to generate quantile coefficients based on the discrete coefficients of the dataset to be checked. The verification result generation module is used to perform box plot verification on the dataset to be verified based on the quantile coefficient, and obtain the verification result of the dataset to be verified.
[0046] Based on the same concept, the present invention also provides a computer device, which may be a terminal, a laptop computer, a desktop computer, a server, a computer cluster, or other types of computer devices.
[0047] Reference Figure 4 A computer device may include at least one processor and memory. The processor can execute instructions stored in the memory. The processor is communicatively connected to the memory via a data bus. In addition to memory, the processor can also be communicatively connected to input devices, output devices, and communication devices via the data bus.
[0048] The processor can be any conventional processor. Processors may include central processing units (CPUs), graphics processing units (GPUs), field-programmable gate arrays (FPGAs), systems on chips (SoCs), application-specific integrated circuits (ASICs), or combinations thereof.
[0049] Memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.
[0050] In this embodiment of the invention, an executable instruction is stored in a memory. The processor can read the executable instruction from the memory and execute the instruction to implement all or part of the steps of the method of the invention.
[0051] Based on the same concept, the present invention also provides a computer program product or a computer-readable storage medium storing the computer program product. The computer product includes computer program instructions that can be executed by a processor to perform all or part of the steps described in the exemplary embodiments above.
[0052] Computer program products can be written in any combination of one or more programming languages to perform the operations of the embodiments of this application. These programming languages include object-oriented programming languages such as Java and C++, as well as conventional procedural programming languages such as C or similar languages, and scripting languages (e.g., Python). The program code can be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0053] Computer-readable storage media can take the form of any combination of one or more readable media. A readable medium can be a readable signal medium or a readable storage medium. A readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: static random access memory (SRAM) having one or more electrically connected wires; electrically erasable programmable read-only memory (EEPROM); erasable programmable read-only memory (EPROM); programmable read-only memory (PROM); read-only memory (ROM); magnetic storage; flash memory; magnetic disk or optical disk; or any suitable combination thereof.
[0054] Although specific embodiments of the invention have been disclosed for illustrative purposes to aid in understanding and implementing the invention, those skilled in the art will understand that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, the invention should not be limited to the content disclosed in the preferred embodiments, and the scope of protection claimed by the invention is defined by the claims.
Claims
1. A dataset-assisted verification method based on improved boxplots, characterized in that, The method includes: Generate a dataset to be verified; Generate quantile coefficients based on the coefficients of dispersion of the dataset to be checked; Based on the quantile coefficient, box plot verification is performed on the dataset to be verified to obtain the verification result of the dataset to be verified.
2. The method according to claim 1, characterized in that, Generate the dataset to be verified, including: Extract the quantitative attribute data from the original dataset; The quantitative attribute data is sorted in ascending order to obtain the dataset to be verified.
3. The method according to claim 1, characterized in that, Based on the coefficients of dispersion of the dataset to be verified, generate quantile coefficients, including: Calculate the average value of the dataset to be verified; Calculate the standard deviation of the dataset to be verified; Calculate the coefficient of variation of the dataset to be checked based on the mean and standard deviation of the data. Based on this coefficient of variation, the quantile coefficients are obtained.
4. The method according to claim 1, characterized in that, Based on the quantile coefficient, boxplot verification is performed on the dataset to be verified, yielding the verification results for the dataset, including: Calculate the upper and lower quartile positions based on the amount of data in the dataset to be verified; Calculate the upper quartile and lower quartile based on their respective positions. Calculate the upper and lower whiskers based on the quantile coefficients, the upper quartile, and the lower quartile; Based on the upper and lower requirements, abnormal data is investigated to obtain the verification results of the dataset to be verified.
5. A dataset-assisted verification device based on improved boxplots, characterized in that, The device includes: The dataset generation module is used to generate the dataset to be verified. The quantile coefficient generation module is used to generate quantile coefficients based on the discrete coefficients of the dataset to be checked. The verification result generation module is used to perform box plot verification on the dataset to be verified based on the quantile coefficient, and obtain the verification result of the dataset to be verified.
6. A computer device, characterized in that, The computer device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the dataset-assisted verification method based on the improved box plot as described in any one of claims 1-4.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the dataset-assisted verification method based on the improved boxplot as described in any one of claims 1-4.
8. A computer program product, characterized in that, When the computer program product is run on a computer device, the computer device performs the dataset-assisted verification method based on the improved box plot as described in any one of claims 1-4.