Data screening method and system based on big data and storage medium

Abstract

The invention discloses a data screening method and system based on big data and a storage medium, and relates to the technical field of data screening, and the method comprises the steps: obtaining to-be-screened data information, obtaining data screening format information based on a data screening demand, screening the to-be-screened data according to the data screening format information, and obtaining initial data information, and obtaining target data information based on data cleaning according to the initial data information. According to the method, clustering analysis is carried out on the initial data through the data clustering matrix, the initial offset data are selected, the initial offset data of different data clustering criteria are compared, data cleaning is achieved, the data screening efficiency is improved, the stability and reliability of the data are ensured, and through the feature correlation coefficient and the feature correlation coefficient threshold value, the accuracy of data screening is improved. Whether the target data features are available or not is judged, the data screening efficiency is improved, and the accuracy and stability of data screening are ensured through secondary data screening condition information.

Description

Technical Field

[0001] The present invention relates to the field of data screening technology, and in particular to a data screening method, system and storage medium based on big data. Background Art

[0002] With the development of the big data environment, data is rapidly accumulated. It is very important to analyze the value contained in massive data and screen out valuable data. It can be seen that data screening plays a vital role in the entire data processing process. The development of big data platforms also provides a technical basis for data screening. Screening usually uses the system to perform conditional and selective screening operations on data. Through data screening, data that meets predetermined conditions is obtained to meet user needs. However, due to the large amount of data for ordinary screening and the fact that screening conditions are manually formulated, errors such as screening conflicts will occur, resulting in frequent incorrect screening and serious impact on screening efficiency.

[0003] At present, there are still problems in data screening, such as data cleaning of the initial data. When removing abnormal data, the data is only cleaned through data outliers or some threshold conditions, which often removes valid data. However, if data cleaning is not performed, the amount of data will be huge, affecting the screening efficiency. At the same time, when screening data, if the data is screened according to too many screening conditions, while ensuring the accuracy of the data, the amount of data obtained will also be reduced. However, if there are too few screening conditions, the data cannot be screened accurately, affecting the stability of the data. Summary of the invention

[0004] In order to solve the above technical problems, a data screening method, system and storage medium based on big data are provided. The technical solution solves the problem of data cleaning of initial data proposed in the above background technology. When removing abnormal data, the data is cleaned only through data outliers or some threshold conditions, which often removes valid data. However, if data cleaning is not performed, it will lead to a huge amount of data, affecting the screening efficiency. At the same time, when screening data, if the data is screened according to too many screening conditions, while ensuring the accuracy of the data, the amount of data obtained is also reduced. However, if there are too few screening conditions, the data cannot be accurately screened, affecting the stability of the data.

[0005] In order to achieve the above purpose, the technical solution adopted by the present invention is: Data screening methods based on big data include: Acquire the data information to be screened, wherein the data information to be screened includes the attribute information of the data to be screened and the format information of the data to be screened; Based on the data screening requirement, acquiring data screening format information, wherein the data screening format information indicates data format restriction information; According to the data screening format information, the data to be screened is screened to obtain the initial data information; According to the initial data information, based on data cleaning, obtain the target data information; Acquire data screening condition information, wherein the data screening condition information includes some data screening feature information; According to the target data information and data screening condition information, based on the chi-square test method, the target data calibration feature information is obtained; According to the target data calibration feature information, secondary data screening condition information is obtained; According to the data filtering condition information and the secondary data filtering condition information, the target data is filtered to obtain the filtered result data.

[0006] Preferably, the step of obtaining target data information based on data cleaning according to the initial data information specifically includes: According to the initial data information, initial data feature information is obtained; According to the initial data feature information and based on the data clustering analysis requirements, obtain the data clustering criterion information; According to the initial data information and the data clustering criterion information, the initial clustering center information corresponding to each data clustering criterion is obtained; According to the initial data information, the initial deduplication data is obtained based on the hash deduplication method; According to the initial deduplication data and the initial cluster center information, based on data clustering, a data clustering matrix is ​​obtained, wherein the data clustering matrix represents the data clustering information corresponding to each data clustering criterion; In the formula, is the bth data cluster under the ath data clustering criterion, r is the total number of data clustering criterion types, is the total number of data clusters under the a-th data clustering criterion; According to the data clustering matrix, the initial offset data corresponding to each data clustering criterion is obtained; According to the initial offset data, the target data information is obtained.

[0007] Preferably, the step of acquiring target data information according to the initial offset data specifically includes: Under the data clustering criterion, if the distance between the initial data and the initial cluster center of the data cluster corresponding to the initial data is greater than the average distance of the data cluster, that is: Then the initial data is used as the initial offset data of the data cluster corresponding to the initial data, where: Indicates the xth initial data The initial cluster center of the corresponding data cluster The distance represents the jth data of the data cluster, and m is the initial data The total number of data in the data cluster to which it belongs; The set of initial offset data of all data clusters is used as the initial offset data of the data clustering criterion to obtain the offset data set ; Based on the data clustering criterion information, the offset data sets of all data clustering criteria are compared, and the non-repeated parts in the offset data sets are used as the initial difference data; The initial difference data in the initial data is removed to obtain the target data information.

[0008] Preferably, the obtaining of initial cluster center information corresponding to each data clustering criterion according to the initial data information and the data clustering criterion information specifically includes: S100: According to the initial data information, arbitrarily select an initial data as the first data centroid; S200: Calculate the distance between the initial data and the centroid of the first data based on the data clustering criterion, and obtain first distance information of the initial data; S300: taking the initial data having the largest first distance from the initial data of the first data centroid as the second data centroid; S400: Obtaining distance information between each initial data and the nearest data centroid; S500: obtaining a data distance difference coefficient based on distance information between the initial data and the nearest data centroid; S600: If the data distance difference coefficient of the initial data , then the initial data is used as the centroid of the candidate data to obtain the centroid set of the candidate data; According to the data centroid set to be selected, the data centroid quality coefficient corresponding to each initial data is obtained; The initial data with the largest data centroid quality coefficient is used as the next data centroid, and steps S400-S600 are repeated; If the maximum value of the data distance difference coefficient , then proceed to step S700; S700: taking the data centroid as the data clustering center and obtaining initial clustering center information; The calculation formula of the data distance difference coefficient is: In the formula, is the data distance difference coefficient of the x-th initial data, represents the distance between the xth initial data and the cth data centroid, A is the data centroid set, represents the first data centroid, represents the second data centroid; The data centroid quality coefficient is specifically: In the formula, is the data centroid quality coefficient of the x-th initial data, It represents the distance between the i-th initial data and the c-th data centroid in the data centroid set to be selected, and n is the total number of initial data in the data centroid set to be selected.

[0009] Preferably, the step of obtaining the secondary data screening condition information according to the target data calibration feature information specifically includes: According to the data screening condition information, obtain the data screening feature information; Based on the data screening features, the target data is classified to obtain the target data information corresponding to each data screening feature; Based on the target data information, the data screening feature with the largest total number of target data is used as the first standard feature, and the data screening feature with the smallest total number of target data is used as the second standard feature; According to the target data information, acquiring target data characteristic information; According to the target data feature information, the first standard feature and the second standard feature, a feature correlation coefficient is obtained based on a chi-square test; Based on the chi-square test analysis, the feature correlation coefficient threshold is obtained; According to the feature correlation coefficient and the feature correlation coefficient threshold, it is determined whether the target data feature is available. If so, the target data feature is used as the target data calibration feature to obtain the target data calibration feature information; Among them, if the characteristic correlation coefficient , then the target data feature is available, if , then the target data feature is not available; According to the target data calibration feature information, secondary data screening condition information is obtained; The calculation formula of the characteristic correlation coefficient is: In the formula, is the characteristic correlation coefficient, is the total number of data for the sth feature combination, is the expected total number of data for the sth feature combination, is the total number of data for the first feature combination, is the total number of data for the second feature combination, It represents the total number of data for the first and second feature combinations.

[0010] Furthermore, a data screening system based on big data is proposed, which is used to implement the above-mentioned data screening method, including: A main control module, the main control module is used to obtain initial offset data corresponding to each data clustering criterion according to a data clustering matrix, determine whether the target data feature is available according to a feature correlation coefficient and a feature correlation coefficient threshold, obtain data clustering criterion information according to initial data feature information and data clustering analysis requirements, obtain initial clustering center information corresponding to each data clustering criterion according to initial data information and data clustering criterion information, obtain a data clustering matrix based on data clustering according to initial deduplication data and initial clustering center information, compare offset data sets of all data clustering criteria based on data clustering criterion information, use a non-repeated part in the offset data set as initial difference data, remove initial difference data in the initial data, obtain target data information, obtain secondary data screening condition information according to target data calibration feature information, screen the target data according to the data screening condition information and secondary data screening condition information, and obtain screened result data; An information acquisition module, the information acquisition module is used to acquire information of data to be screened, attribute information of data to be screened and format information of data to be screened, acquire data screening format information based on data screening requirements, screen the data to be screened according to the data screening format information, acquire initial data information, acquire initial data feature information according to the initial data information, and acquire data screening feature information according to the data screening condition information; An evaluation module, wherein the evaluation module is used to obtain a data distance difference coefficient based on the distance information between the initial data and the nearest data centroid, obtain a data centroid quality coefficient corresponding to each initial data according to the selected data centroid set, and obtain a feature correlation coefficient based on the chi-square test according to the target data feature information, the first standard feature and the second standard feature; A display module interacts with the main control module and is used to display target data information, data screening condition information, secondary data screening condition information and result data.

[0011] Optionally, the main control module specifically includes: A control unit, the control unit is used to obtain data clustering criterion information based on initial data feature information and data clustering analysis requirements, obtain initial clustering center information corresponding to each data clustering criterion based on the initial data information and the data clustering criterion information, obtain a data clustering matrix based on data clustering according to the initial deduplication data and the initial clustering center information, compare the offset data sets of all data clustering criteria based on the data clustering criterion information, use the non-repeated part in the offset data set as the initial difference data, remove the initial difference data in the initial data, obtain target data information, obtain secondary data screening condition information based on target data calibration feature information, screen the target data based on the data screening condition information and the secondary data screening condition information, and obtain screened result data; An information receiving unit, which interacts with the information acquisition module and the evaluation module to receive data and transmit it to the judgment unit; The judgment unit is used to obtain initial offset data corresponding to a data clustering criterion according to a data clustering matrix, and to judge whether the target data feature is available according to a feature correlation coefficient and a feature correlation coefficient threshold.

[0012] Optionally, the information acquisition module specifically includes: A first acquisition unit, the first acquisition unit is used to acquire the to-be-screened data information, the to-be-screened data attribute information and the to-be-screened data format information, and acquire the data screening format information based on the data screening requirements; The second acquisition unit is used to filter the data to be filtered according to the data filtering format information, obtain the initial data information, obtain the initial data feature information according to the initial data information, and obtain the data filtering feature information according to the data filtering condition information.

[0013] Optionally, the evaluation module specifically includes: A first evaluation unit, the first evaluation unit is used to obtain a data distance difference coefficient based on the distance information between the initial data and the nearest data centroid, and obtain a data centroid quality coefficient corresponding to each initial data according to the data centroid set to be selected; The second evaluation unit is used to obtain a feature correlation coefficient based on a chi-square test according to the target data feature information, the first standard feature and the second standard feature.

[0014] Furthermore, a computer-readable storage medium is provided, on which a computer-readable program is stored, and when the computer-readable program is called, the data screening method as described above is executed.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The present invention proposes a data screening method, system and storage medium based on big data, performs cluster analysis on initial data through a data clustering matrix, selects initial offset data, and achieves data cleaning by comparing initial offset data of different data clustering criteria, improves data screening efficiency, ensures data stability and reliability, and determines whether the target data feature is available through feature correlation coefficients and feature correlation coefficient thresholds, thereby improving data screening efficiency, and ensures data screening accuracy and stability through secondary data screening condition information. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of the data screening method based on big data proposed by the present invention; Figure 2 This is a flow chart for obtaining target data information in the present invention; Figure 3 This is a flow chart for obtaining the initial cluster center information in the present invention; Figure 4 This is a flow chart for obtaining secondary data screening condition information in the present invention; Figure 5 This is a structural block diagram of the data screening system based on big data proposed in the present invention. DETAILED DESCRIPTION

[0017] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art may think of other obvious variations.

[0018] Reference Figure 1 - Figure 4 As shown, the data screening method based on big data in the embodiment of the present invention includes: Acquire the data information to be screened, wherein the data information to be screened includes the attribute information of the data to be screened and the format information of the data to be screened; Based on the data screening requirement, acquiring data screening format information, wherein the data screening format information indicates data format restriction information; According to the data screening format information, the data to be screened is screened to obtain the initial data information; According to the initial data information, based on data cleaning, obtain the target data information; Specifically, according to the initial data information, based on data cleaning, the target data information is obtained, including: According to the initial data information, initial data feature information is obtained; According to the initial data feature information and based on the data clustering analysis requirements, obtain the data clustering criterion information; According to the initial data information and the data clustering criterion information, the initial clustering center information corresponding to each data clustering criterion is obtained; According to the initial data information, the initial deduplication data is obtained based on the hash deduplication method; According to the initial deduplication data and the initial cluster center information, based on data clustering, a data clustering matrix is ​​obtained, wherein the data clustering matrix represents the data clustering information corresponding to each data clustering criterion; In the formula, is the bth data cluster under the ath data clustering criterion, r is the total number of data clustering criterion types, is the total number of data clusters under the a-th data clustering criterion; According to the data clustering matrix, the initial offset data corresponding to each data clustering criterion is obtained; According to the initial offset data, the target data information is obtained.

[0019] Specifically, according to the initial data information and the data clustering criterion information, the initial clustering center information corresponding to each data clustering criterion is obtained, which specifically includes: S100: According to the initial data information, arbitrarily select an initial data as the first data centroid; S200: Calculate the distance between the initial data and the centroid of the first data based on the data clustering criterion, and obtain first distance information of the initial data; S300: taking the initial data having the largest first distance from the initial data of the first data centroid as the second data centroid; S400: Obtaining distance information between each initial data and the nearest data centroid; S500: obtaining a data distance difference coefficient based on distance information between the initial data and the nearest data centroid; S600: If the data distance difference coefficient of the initial data , then the initial data is used as the centroid of the candidate data to obtain the centroid set of the candidate data; According to the data centroid set to be selected, the data centroid quality coefficient corresponding to each initial data is obtained; The initial data with the largest data centroid quality coefficient is used as the next data centroid, and steps S400-S600 are repeated; If the maximum value of the data distance difference coefficient , then proceed to step S700; S700: taking the data centroid as the data clustering center and obtaining initial clustering center information; The calculation formula of the data distance difference coefficient is: In the formula, is the data distance difference coefficient of the x-th initial data, represents the distance between the xth initial data and the cth data centroid, A is the data centroid set, represents the first data centroid, represents the second data centroid; The data centroid quality coefficient is specifically: In the formula, is the data centroid quality coefficient of the x-th initial data, It represents the distance between the i-th initial data and the c-th data centroid in the data centroid set to be selected, and n is the total number of initial data in the data centroid set to be selected.

[0020] In this scheme, an initial data is arbitrarily selected as the first data centroid, the distance between the initial data and the first data centroid is calculated, and the initial data with the largest first distance to the initial data of the first data centroid is used as the second data centroid. Based on the distance information between the initial data and the nearest data centroid, the data distance difference coefficient is obtained. Through the data distance difference coefficient, it is determined whether the initial data can be used as the data centroid. By using the initial data with the largest data centroid quality coefficient as the next data centroid, the accuracy of the data clustering center is ensured and the accuracy of data clustering is improved.

[0021] Specifically, according to the initial offset data, the target data information is obtained, including: Under the data clustering criterion, if the distance between the initial data and the initial cluster center of the data cluster corresponding to the initial data is greater than the average distance of the data cluster, that is: Then the initial data is used as the initial offset data of the data cluster corresponding to the initial data, where: Indicates the xth initial data The initial cluster center of the corresponding data cluster The distance represents the jth data of the data cluster, and m is the initial data The total number of data in the data cluster to which it belongs; The set of initial offset data of all data clusters is used as the initial offset data of the data clustering criterion to obtain the offset data set ; Based on the data clustering criterion information, the offset data sets of all data clustering criteria are compared, and the non-repeated parts in the offset data sets are used as the initial difference data; The initial difference data in the initial data is removed to obtain the target data information.

[0022] In this scheme, the initial offset data is selected by comparing the distance between the initial data and the initial cluster center of the data cluster corresponding to the initial data and the average distance of the data cluster. The offset data of each data cluster are aggregated to obtain the offset data set corresponding to each data clustering criterion. All the offset data sets are then compared to remove the non-duplicate data therein. It should be noted that in this embodiment, non-repeated data refers to data that does not appear in the offset data set corresponding to each data clustering criterion at the same time. The data clustering criterion is the characteristic item of the data, such as time, data value, etc. The distance of the data is the data difference status calculated based on the data clustering criterion.

[0023] Acquire data screening condition information, wherein the data screening condition information includes some data screening feature information; According to the target data information and data screening condition information, based on the chi-square test method, the target data calibration feature information is obtained; According to the target data calibration feature information, secondary data screening condition information is obtained; Specifically, according to the target data calibration feature information, the secondary data screening condition information is obtained, including: According to the data screening condition information, obtain the data screening feature information; Based on the data screening features, the target data is classified to obtain the target data information corresponding to each data screening feature; Based on the target data information, the data screening feature with the largest total number of target data is used as the first standard feature, and the data screening feature with the smallest total number of target data is used as the second standard feature; According to the target data information, acquiring target data characteristic information; According to the target data feature information, the first standard feature and the second standard feature, a feature correlation coefficient is obtained based on a chi-square test; Based on the chi-square test analysis, the feature correlation coefficient threshold is obtained; According to the feature correlation coefficient and the feature correlation coefficient threshold, it is determined whether the target data feature is available. If so, the target data feature is used as the target data calibration feature to obtain the target data calibration feature information; Among them, if the characteristic correlation coefficient , then the target data feature is available, if , then the target data feature is not available; According to the target data calibration feature information, secondary data screening condition information is obtained; The calculation formula of the characteristic correlation coefficient is: In the formula, is the characteristic correlation coefficient, is the total number of data for the sth feature combination, is the expected total number of data for the sth feature combination, is the total number of data for the first feature combination, is the total number of data for the second feature combination, It represents the total number of data for the first and second feature combinations.

[0024] In this solution, the target data is classified based on the data screening features, the target data information corresponding to each data screening feature is obtained, the data screening feature with the largest total number of target data is used as the first standard feature, and the data screening feature with the smallest total number of target data is used as the second standard feature. By taking the target data feature information, the first standard feature and the second standard feature, based on the chi-square test, the feature correlation coefficient is obtained, and the feature correlation coefficient and the feature correlation coefficient threshold are used to determine whether the target data feature is available, and the target data calibration feature information is obtained. According to the target data calibration feature information, the secondary data screening condition information is obtained, and the accuracy and stability of data screening are ensured through the secondary data screening condition information.

[0025] In this embodiment, the total number of data of the first feature combination is data having both the target data feature and the first standard feature, and the total number of data of the second feature combination is data having both the target data feature and the second standard feature. It represents the combined total number of data of the first feature combination and the second feature combination, that is, the total number of data of the first feature combination and the second feature combination. It should be noted that the total number of data here refers to the total number of data after removing the number of duplicate data, rather than the sum of the total number of data of the first feature combination and the total number of data of the second feature combination.

[0026] It is understandable that features can be selected directly through screening conditions (such as selecting values ​​within a certain range, filtering by certain statistical standards, etc.). For example, you can filter out features with variance greater than a certain value, but this screening method ignores the relationship between the feature and the target variable, and does not consider the mutual influence between variables. Through the feature correlation coefficient, data features without set screening conditions are selected, features are calibrated based on the target data, and secondary data screening conditions are set, which improves the accuracy of data screening.

[0027] According to the data filtering condition information and the secondary data filtering condition information, the target data is filtered to obtain the filtered result data.

[0028] Reference Figure 5 As shown, further, in combination with the above-mentioned data screening method based on big data, a data screening system based on big data is proposed, including: A main control module, the main control module is used to obtain initial offset data corresponding to each data clustering criterion according to a data clustering matrix, determine whether the target data feature is available according to a feature correlation coefficient and a feature correlation coefficient threshold, obtain data clustering criterion information according to initial data feature information and data clustering analysis requirements, obtain initial clustering center information corresponding to each data clustering criterion according to initial data information and data clustering criterion information, obtain a data clustering matrix based on data clustering according to initial deduplication data and initial clustering center information, compare offset data sets of all data clustering criteria based on data clustering criterion information, use a non-repeated part in the offset data set as initial difference data, remove initial difference data in the initial data, obtain target data information, obtain secondary data screening condition information according to target data calibration feature information, screen the target data according to the data screening condition information and secondary data screening condition information, and obtain screened result data; An information acquisition module, the information acquisition module is used to acquire information of data to be screened, attribute information of data to be screened and format information of data to be screened, acquire data screening format information based on data screening requirements, screen the data to be screened according to the data screening format information, acquire initial data information, acquire initial data feature information according to the initial data information, and acquire data screening feature information according to the data screening condition information; An evaluation module, wherein the evaluation module is used to obtain a data distance difference coefficient based on the distance information between the initial data and the nearest data centroid, obtain a data centroid quality coefficient corresponding to each initial data according to the selected data centroid set, and obtain a feature correlation coefficient based on the chi-square test according to the target data feature information, the first standard feature and the second standard feature; A display module interacts with the main control module and is used to display target data information, data screening condition information, secondary data screening condition information and result data.

[0029] Main control module, specifically including: A control unit, the control unit is used to obtain data clustering criterion information based on initial data feature information and data clustering analysis requirements, obtain initial clustering center information corresponding to each data clustering criterion based on the initial data information and the data clustering criterion information, obtain a data clustering matrix based on data clustering according to the initial deduplication data and the initial clustering center information, compare the offset data sets of all data clustering criteria based on the data clustering criterion information, use the non-repeated part in the offset data set as the initial difference data, remove the initial difference data in the initial data, obtain target data information, obtain secondary data screening condition information based on target data calibration feature information, screen the target data based on the data screening condition information and the secondary data screening condition information, and obtain screened result data; An information receiving unit, which interacts with the information acquisition module and the evaluation module to receive data and transmit it to the judgment unit; The judgment unit is used to obtain initial offset data corresponding to a data clustering criterion according to a data clustering matrix, and to judge whether the target data feature is available according to a feature correlation coefficient and a feature correlation coefficient threshold.

[0030] Information acquisition module, specifically including: A first acquisition unit, the first acquisition unit is used to acquire the to-be-screened data information, the to-be-screened data attribute information and the to-be-screened data format information, and acquire the data screening format information based on the data screening requirements; The second acquisition unit is used to filter the data to be filtered according to the data filtering format information, obtain the initial data information, obtain the initial data feature information according to the initial data information, and obtain the data filtering feature information according to the data filtering condition information.

[0031] Assessment modules include: A first evaluation unit, the first evaluation unit is used to obtain a data distance difference coefficient based on the distance information between the initial data and the nearest data centroid, and obtain a data centroid quality coefficient corresponding to each initial data according to the data centroid set to be selected; The second evaluation unit is used to obtain a feature correlation coefficient based on a chi-square test according to the target data feature information, the first standard feature and the second standard feature.

[0032] It is understandable that the computer-readable storage medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; an optical medium, such as a DVD; or a semiconductor medium, such as a solid state disk (SSD).

[0033] In summary, the advantages of the present invention are: based on the data clustering analysis requirements, data clustering criterion information is obtained through initial data feature information; initial clustering center information corresponding to each data clustering criterion is obtained through initial data information and data clustering criterion information; based on data clustering, a data clustering matrix is ​​obtained through initial deduplication data and initial clustering center information; cluster analysis is performed on the initial data through the data clustering matrix to select initial offset data; by comparing the initial offset data of different data clustering criteria, non-repeated data is removed to obtain target data, thereby achieving data cleaning, improving data screening efficiency, and ensuring data stability and reliability; based on the target data feature information, the first standard feature and the second standard feature, a feature correlation coefficient is obtained based on the chi-square test; through the feature correlation coefficient and the feature correlation coefficient threshold, whether the target data feature is available is determined; target data calibration feature information is obtained; according to the target data calibration feature information, secondary data screening condition information is obtained; and the accuracy and stability of data screening are ensured through the secondary data screening condition information.

[0034] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A data screening method based on big data, characterized in that: include: Acquire the data information to be screened, wherein the data information to be screened includes the attribute information of the data to be screened and the format information of the data to be screened; Based on the data screening requirement, acquiring data screening format information, wherein the data screening format information indicates data format restriction information; According to the data screening format information, the data to be screened is screened to obtain the initial data information; According to the initial data information, based on data cleaning, obtain the target data information; Acquire data screening condition information, wherein the data screening condition information includes some data screening feature information; According to the target data information and data screening condition information, based on the chi-square test method, the target data calibration feature information is obtained; According to the target data calibration feature information, secondary data screening condition information is obtained; Filter the target data according to the data filtering condition information and the secondary data filtering condition information, and obtain the filtered result data; The obtaining of secondary data screening condition information according to the target data calibration feature information specifically includes: According to the data screening condition information, obtain the data screening feature information; Based on the data screening features, the target data is classified to obtain the target data information corresponding to each data screening feature; Based on the target data information, the data screening feature with the largest total number of target data is used as the first standard feature, and the data screening feature with the smallest total number of target data is used as the second standard feature; According to the target data information, acquiring target data characteristic information; According to the target data feature information, the first standard feature and the second standard feature, a feature correlation coefficient is obtained based on a chi-square test; Based on the chi-square test analysis, the feature correlation coefficient threshold is obtained; According to the feature correlation coefficient and the feature correlation coefficient threshold, it is determined whether the target data feature is available. If so, the target data feature is used as the target data calibration feature to obtain the target data calibration feature information; Among them, if the characteristic correlation coefficient , then the target data feature is available, if , then the target data feature is not available; According to the target data calibration feature information, secondary data screening condition information is obtained; The calculation formula of the characteristic correlation coefficient is: In the formula, is the characteristic correlation coefficient, is the total number of data for the sth feature combination, is the expected total number of data for the sth feature combination, is the total number of data for the first feature combination, is the total number of data for the second feature combination, It represents the total number of data for the first and second feature combinations.

2. The data screening method based on big data according to claim 1, characterized in that: The step of obtaining target data information based on data cleaning according to the initial data information specifically includes: According to the initial data information, initial data feature information is obtained; According to the initial data feature information and based on the data clustering analysis requirements, obtain the data clustering criterion information; According to the initial data information and the data clustering criterion information, the initial clustering center information corresponding to each data clustering criterion is obtained; According to the initial data information, the initial deduplication data is obtained based on the hash deduplication method; According to the initial deduplication data and the initial cluster center information, based on data clustering, a data clustering matrix is ​​obtained, wherein the data clustering matrix represents the data clustering information corresponding to each data clustering criterion; In the formula, is the bth data cluster under the ath data clustering criterion, r is the total number of data clustering criterion types, is the total number of data clusters under the a-th data clustering criterion; According to the data clustering matrix, the initial offset data corresponding to each data clustering criterion is obtained; According to the initial offset data, the target data information is obtained.

3. The data screening method based on big data according to claim 2 is characterized in that: The step of obtaining target data information according to the initial offset data specifically includes: Under the data clustering criterion, if the distance between the initial data and the initial cluster center of the data cluster corresponding to the initial data is greater than the average distance of the data cluster, that is: Then the initial data is used as the initial offset data of the data cluster corresponding to the initial data, where: Indicates the xth initial data The initial cluster center of the corresponding data cluster The distance represents the jth data of the data cluster, and m is the initial data The total number of data in the data cluster to which it belongs; The set of initial offset data of all data clusters is used as the initial offset data of the data clustering criterion to obtain the offset data set ; Based on the data clustering criterion information, the offset data sets of all data clustering criteria are compared, and the non-repeated parts in the offset data sets are used as the initial difference data; The initial difference data in the initial data is removed to obtain the target data information.

4. The data screening method based on big data according to claim 2, characterized in that: The step of obtaining the initial cluster center information corresponding to each data clustering criterion according to the initial data information and the data clustering criterion information specifically includes: S100: According to the initial data information, arbitrarily select an initial data as the first data centroid; S200: Calculate the distance between the initial data and the centroid of the first data based on the data clustering criterion, and obtain first distance information of the initial data; S300: taking the initial data having the largest first distance from the initial data of the first data centroid as the second data centroid; S400: Obtaining distance information between each initial data and the nearest data centroid; S500: obtaining a data distance difference coefficient based on distance information between the initial data and the nearest data centroid; S600: If the data distance difference coefficient of the initial data , then the initial data is used as the centroid of the candidate data to obtain the centroid set of the candidate data; According to the data centroid set to be selected, the data centroid quality coefficient corresponding to each initial data is obtained; The initial data with the largest data centroid quality coefficient is used as the next data centroid, and steps S400-S600 are repeated; If the maximum value of the data distance difference coefficient , then proceed to step S700; S700: taking the data centroid as the data clustering center and obtaining initial clustering center information; The calculation formula of the data distance difference coefficient is: In the formula, is the data distance difference coefficient of the x-th initial data, represents the distance between the xth initial data and the cth data centroid, A is the data centroid set, represents the first data centroid, represents the second data centroid; The data centroid quality coefficient is specifically: In the formula, is the data centroid quality coefficient of the x-th initial data, It represents the distance between the i-th initial data and the c-th data centroid in the data centroid set to be selected, and n is the total number of initial data in the data centroid set to be selected.

5. A data screening system based on big data, used to implement the data screening method according to any one of claims 1 to 4, characterized in that: include: A main control module, the main control module is used to obtain initial offset data corresponding to each data clustering criterion according to a data clustering matrix, determine whether the target data feature is available according to a feature correlation coefficient and a feature correlation coefficient threshold, obtain data clustering criterion information according to initial data feature information and data clustering analysis requirements, obtain initial clustering center information corresponding to each data clustering criterion according to initial data information and data clustering criterion information, obtain a data clustering matrix based on data clustering according to initial deduplication data and initial clustering center information, compare offset data sets of all data clustering criteria based on data clustering criterion information, use a non-repeated part in the offset data set as initial difference data, remove initial difference data in the initial data, obtain target data information, obtain secondary data screening condition information according to target data calibration feature information, screen the target data according to the data screening condition information and secondary data screening condition information, and obtain screened result data; An information acquisition module, the information acquisition module is used to acquire information of data to be screened, attribute information of data to be screened and format information of data to be screened, acquire data screening format information based on data screening requirements, screen the data to be screened according to the data screening format information, acquire initial data information, acquire initial data feature information according to the initial data information, and acquire data screening feature information according to the data screening condition information; An evaluation module, wherein the evaluation module is used to obtain a data distance difference coefficient based on the distance information between the initial data and the nearest data centroid, obtain a data centroid quality coefficient corresponding to each initial data according to the selected data centroid set, and obtain a feature correlation coefficient based on the chi-square test according to the target data feature information, the first standard feature and the second standard feature; A display module interacts with the main control module and is used to display target data information, data screening condition information, secondary data screening condition information and result data.

6. The data screening system based on big data according to claim 5, characterized in that: The main control module specifically includes: A control unit, the control unit is used to obtain data clustering criterion information based on initial data feature information and data clustering analysis requirements, obtain initial clustering center information corresponding to each data clustering criterion based on the initial data information and the data clustering criterion information, obtain a data clustering matrix based on data clustering according to the initial deduplication data and the initial clustering center information, compare the offset data sets of all data clustering criteria based on the data clustering criterion information, use the non-repeated part in the offset data set as the initial difference data, remove the initial difference data in the initial data, obtain target data information, obtain secondary data screening condition information based on target data calibration feature information, screen the target data based on the data screening condition information and the secondary data screening condition information, and obtain screened result data; An information receiving unit, which interacts with the information acquisition module and the evaluation module to receive data and transmit it to the judgment unit; The judgment unit is used to obtain initial offset data corresponding to a data clustering criterion according to a data clustering matrix, and to judge whether the target data feature is available according to a feature correlation coefficient and a feature correlation coefficient threshold.

7. The data screening system based on big data according to claim 5, characterized in that: The information acquisition module specifically includes: A first acquisition unit, the first acquisition unit is used to acquire the to-be-screened data information, the to-be-screened data attribute information and the to-be-screened data format information, and acquire the data screening format information based on the data screening requirements; The second acquisition unit is used to filter the data to be filtered according to the data filtering format information, obtain the initial data information, obtain the initial data feature information according to the initial data information, and obtain the data filtering feature information according to the data filtering condition information.

8. The data screening system based on big data according to claim 5, characterized in that: The evaluation module specifically includes: A first evaluation unit, the first evaluation unit is used to obtain a data distance difference coefficient based on the distance information between the initial data and the nearest data centroid, and obtain a data centroid quality coefficient corresponding to each initial data according to the data centroid set to be selected; The second evaluation unit is used to obtain a feature correlation coefficient based on a chi-square test according to the target data feature information, the first standard feature and the second standard feature.

9. A computer-readable storage medium having a computer-readable program stored thereon, characterized in that: When the computer-readable program is called, the data screening method according to any one of claims 1 to 4 is executed.

Images