Data screening method and data screening device

Abstract

The invention provides a data screening method and a data screening device. The data screening method comprises the steps of drawing a scatter diagram based on obtained discrete data in a to-be-screened data set; based on the distribution characteristics of each discrete point in the scatter diagram, determining a fitting curve connecting the plurality of discrete points and the fitting degree ofthe fitting curve; if the fitting degree of the fitting curve is smaller than a preset threshold value, constructing a probability distribution model based on the distance from each discrete point tothe fitting curve; determining a confidence interval range of the distance based on the average value and the standard error of the probability distribution model and the obtained saliency level value; and determining discrete data corresponding to the discrete points, of which the distance values with the fitting curve are located outside the confidence interval range, in all the discrete points,screening the determined discrete data from a to-be-screened data set, and determining the data set with the discrete data screened as a target data set. Thus, outliers can be removed with the fitting curve as the reference, and the outliers removal accuracy and thoroughness are improved.

Description

technical field [0001] The present application relates to the technical field of big data processing, in particular to a data screening method and a data screening device. Background technique [0002] With the rapid development of Internet technology, big data technology has penetrated into many businesses in many industries. By collecting a large amount of business-related business data, a large amount of business data is processed and analyzed, and then the industry corresponding to the business data is analyzed. In the collected data, there are often some interference data due to measurement deviation or statistical deviation. These interference data will affect the processing and analysis process of the data, thereby affecting the analysis and judgment of the industry. [0003] At this stage, common methods for removing interference data in large data sets, such as building data models, etc., are all based on predicted values ​​or pre-set standards to remove interferenc...

AI Technical Summary

Problems solved by technology

In the collected data, there are often some interference data due to measurement deviation or statistical deviation. These interference data will affect the processing and analysis of the data, thus affecting the analysis and judgment of the industry.

[0003] At this stage, common methods for removing interference data in large data sets, such as building data models, etc., are all based on predicted values ​​or pre-set standards to remove interference data, and do not use the data to be screened as a benchmark. Due to the different characteristics of the data, Preset standards may not be suitable for all data, resulting in incomplete or inaccurate removal of interference data, affecting data processing and analysis

Images