Data filtering method and device, equipment and computer readable storage medium

Abstract

The invention relates to the technical field of big data, and discloses a data filtering method. The method comprises the following steps: obtaining an original data set, and splitting the original data set into a plurality of sub-data sets; calculating information gain rates of all sub-data in each sub-data set by utilizing a pre-constructed information gain model, and extracting feature data of the corresponding sub-data set according to the information gain rates to obtain a feature data set; and executing data filtering of the feature data set based on a preset data filtering mechanism. The invention also relates to a blockchain technology, and the feature data set can be deployed in a blockchain node. According to the invention, data filtering of mass data can be supported.

Description

technical field [0001] The present invention relates to the technical field of big data, in particular to a data filtering method, device, electronic equipment and computer-readable storage medium. Background technique [0002] Data filtering refers to a method of classifying, cleaning, and filtering a large amount of user data generated on the Internet. The rapid development of the Internet has given birth to the era of big data, and the rise of the mobile Internet has made big data ubiquitous, causing data problems such as excess, confusion, and abuse, causing serious harm and economic losses to users. , data filtering becomes very important. [0003] The current data filtering method is mainly based on some simple data characteristics to achieve data filtering, such as filtering data based on some common keywords, or some commonly used field identifiers. This data filtering method is only suitable for situations with a small amount of data. , when the amount of data dou...

AI Technical Summary

Problems solved by technology

[0003] The current data filtering method is mainly based on some simple data characteristics to achieve data filtering, such as filtering data based on some common keywords, or some commonly used field identifiers. This data filtering method is only suitable for situations with a small amount of data. , when the amount of data doubles, it is easy to cause confusion in keywords or fields. For example, if there are keywords A1 or field A2, data A needs to be filtered, but when there are massive data, keywords also exist in data B A1 or field A2 does not need to filter data B, so the above data filtering method cannot support a large amount of data filtering, which will reduce the quality of data filtering

Images