Data mining system based on big data

Abstract

The invention relates to the technical field of big data mining, and discloses a data mining system based on big data, which comprises a big data database, a data mining system and a mining data processing system. The data mining system is used for mining image data, the data preprocessing system is used for cleaning the mined image data, the data mining algorithm execution system is used for classifying the image data, the data evaluation system is used for evaluating the classified image data, and an interactive interface is generated after the data passes so that the data can be quickly inquired. According to the method, after data classification and evaluation are completed, mined data are expressed in a proper form, the data are stored in a database and transmitted into an enterprise database, collision training is carried out on the data and an image database in the enterprise database, and acquaintance image data are taken out, so that more acquaintance image data are prevented from occurring; and the image database is bloated.

Description

Technical Field

[0001] The present invention relates to the technical field of big data mining, and in particular to a data mining system based on big data. Background Art

[0002] Data mining refers to extracting potential, effective, novel, and useful patterns or information from large amounts of noisy and random data for decision makers to make corresponding decisions. In practice, data mining can help people make judgments so that they can take appropriate actions.

[0003] Existing data mining systems can only perform simple statistical processing on existing data, but cannot perform deeper data mining and analysis based on existing data, making it difficult to fully utilize the data. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a data mining system based on big data, which solves the problems raised in the above background technology.

[0005] To solve the above technical problems, according to one aspect of the present invention, more specifically, a data mining system based on big data includes a big data database, a data mining system and a mining data processing system, which is characterized by: the big data database: a large amount of unstructured or structured data from various sources;

[0006] Data mining system: a system that mines specific image data from a large amount of data stored in a database, data warehouse or other information repository. The data mining system pre-processes the image data, performs mining and data evaluation;

[0007] Mining data processing system: protect the mined image data, evaluate the security of the mined data, and protect the mined image data, including data representation and storage, data security control and data privacy protection.

[0008] Furthermore, the data preprocessing system includes data cleaning, data integration, data conversion and data reduction;

[0009] Data cleaning: used to remove errors, duplications, inconsistencies and noise in corrected data;

[0010] Data integration: combining data from different data sources into a consistent data set;

[0011] Data conversion: normalize, standardize, discretize or encode data to meet the needs of mining algorithms;

[0012] Data reduction: Reduce the amount of data through clustering, sampling and other techniques while trying to preserve the integrity of the original data.

[0013] Furthermore, the data mining algorithm execution system includes data classification, data clustering, association rule mining, anomaly detection and evolution analysis;

[0014] Data classification: classify a dataset into predefined categories and predict the category of new data points;

[0015] Data clustering: Divide a data set into multiple groups or clusters so that the data points within the same cluster are more similar, while the data points between different clusters are less similar;

[0016] Association rule mining: discovering frequent patterns, correlations, or causal relationships between data items;

[0017] Anomaly detection: Identifying data points that are significantly different from the majority of the data, i.e. outliers or outliers;

[0018] Evolutionary analysis: Analyze trends and patterns in data over time.

[0019] Furthermore, the data evaluation system includes a data evaluation and generation interaction interface;

[0020] Data evaluation: Evaluate the mined image data to determine whether it is meaningful and reliable, and use visualization technology to present the mining results to users in the form of charts, graphs, etc. to facilitate user understanding and interpretation;

[0021] Generate interactive interface: Provide a user interactive interface that allows users to adjust mining parameters, select mining algorithms, or view detailed information about the mining process according to their needs.

[0022] Furthermore, the mining data processing system includes a data representation and storage module and a data security and privacy protection module;

[0023] Data representation and storage module: represents the mined image data in an appropriate form and stores the image data in a database for subsequent query, analysis and application;

[0024] Data security and privacy protection module: used to ensure the security and privacy of image data during data mining, prevent image data leakage or illegal use, and provide security measures such as data encryption and access control to protect user data security.

[0025] Furthermore, when the data mining algorithm performs data classification in the system,

[0026]

[0027] In the formula, G A(D) is the similarity between any two subsets in the data classification set; D is the total set of data classification sets; D1 and D2 are both subsets of the data classification set; P(D) is the prior probability of the data classification set of category D; P(x i / D) is the conditional probability of feature i under the condition of category D; P(xi) is the joint probability of i in all features D;

[0028] The data obtained from the formula takes values ​​from 0 to 1:

[0029] 0 means exactly the same;

[0030] 1 means completely different. When the formula structure tends to 0, the similarity between the two subsets increases. When the formula structure tends to 1, the similarity between the two subsets decreases.

[0031] A method for using a data mining system based on big data comprises the following steps:

[0032] S1. Connecting to the big data database: connecting the data mining system to the big data database so that the system can mine big data;

[0033] S2. Data processing: mining the big data database through the data preprocessing system, and cleaning, integrating, converting and simplifying the mined image data;

[0034] S3, Mining Execution: Classify the preprocessed image data, discover frequent patterns, correlations or causal relationships between data items, and analyze the trends and patterns of data changes over time;

[0035] S4, Data evaluation: Evaluate the mined image data to determine whether it is meaningful and reliable, and use visualization technology to display the mining results to users in the form of charts, graphs, etc., so that users can understand and interpret them, and generate interactive interfaces for query;

[0036] S5. Mining data processing: protecting the mined image data, evaluating the security of the mined data, and protecting the mined data;

[0037] S6. Connect to the enterprise database: transfer the image data from the mining data processing system to the enterprise database, perform collision training with the image database inside the enterprise database, and extract the familiar image data to avoid too much familiar image data, which will cause the image database to be bloated.

[0038] The beneficial effects of the data mining system based on big data of the present invention are:

[0039] The present invention mines a big data database through a data mining system, cleans the mined image data through a data preprocessing system, and uses a data mining algorithm execution system to classify the image data, and then uses a data evaluation system to evaluate the classified image data, and then generates an interactive interface to quickly query this type of data. After the data classification and evaluation are completed, the present invention presents the mined data in an appropriate form, stores the data in a database, and transmits it to the enterprise database for collision training with the image database in the enterprise database, and takes out the familiar image data to avoid the appearance of more familiar image data, which leads to a bloated image database. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0041] Figure 1 It is a structural schematic diagram of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0043] like Figure 1 As shown, according to one aspect of the present invention, a data mining system based on big data is provided, including a big data database, a data mining system and a mining data processing system, characterized in that: the big data database: a large amount of unstructured or structured data from various sources;

[0044] Data mining system: a system that mines specific image data from a large amount of data stored in a database, data warehouse or other information repository. The data mining system pre-processes the image data, performs mining and data evaluation;

[0045] Mining data processing system: protect the mined image data, evaluate the security of the mined data, and protect the mined image data, including data representation and storage, data security control and data privacy protection.

[0046] In this embodiment, the data preprocessing system includes data cleaning, data integration, data conversion and data reduction;

[0047] Data cleaning: used to remove errors, duplications, inconsistencies and noise in corrected data;

[0048] Data integration: combining data from different data sources into a consistent data set;

[0049] Data conversion: normalize, standardize, discretize or encode data to meet the needs of mining algorithms;

[0050] Data reduction: Reduce the amount of data through clustering, sampling and other techniques while trying to preserve the integrity of the original data.

[0051] In this embodiment, the data mining algorithm execution system includes data classification, data clustering, association rule mining, anomaly detection and evolution analysis;

[0052] Data classification: classify a dataset into predefined categories and predict the category of new data points;

[0053] Data clustering: Divide a data set into multiple groups or clusters so that the data points within the same cluster are more similar, while the data points between different clusters are less similar;

[0054] Association rule mining: discovering frequent patterns, correlations, or causal relationships between data items;

[0055] Anomaly detection: Identifying data points that are significantly different from the majority of the data, i.e. outliers or outliers;

[0056] Evolutionary analysis: Analyze trends and patterns in data over time.

[0057] In this embodiment, the data evaluation system includes a data evaluation and generation interaction interface;

[0058] Data evaluation: Evaluate the mined image data to determine whether it is meaningful and reliable, and use visualization technology to present the mining results to users in the form of charts, graphs, etc. to facilitate user understanding and interpretation;

[0059] Generate interactive interface: Provide a user interactive interface that allows users to adjust mining parameters, select mining algorithms, or view detailed information about the mining process according to their needs.

[0060] In this embodiment, the mining data processing system includes a data representation and storage module and a data security and privacy protection module;

[0061] Data representation and storage module: represents the mined image data in an appropriate form and stores the image data in a database for subsequent query, analysis and application;

[0062] Data security and privacy protection module: used to ensure the security and privacy of image data during data mining, prevent image data leakage or illegal use, and provide security measures such as data encryption and access control to protect user data security.

[0063] In this embodiment, when the data mining algorithm performs data classification in the system,

[0064]

[0065] In the formula, G A (D) is the similarity between any two subsets in the data classification set; D is the total set of data classification sets; D1 and D2 are both subsets of the data classification set; P(D) is the prior probability of the data classification set of category D; P(x i / D) is the conditional probability of feature i under the condition of category D; P(xi) is the joint probability of i in all features D;

[0066] The data obtained from the formula takes values ​​from 0 to 1:

[0067] 0 means exactly the same;

[0068] 1 means completely different. When the formula structure tends to 0, the similarity between the two subsets increases. When the formula structure tends to 1, the similarity between the two subsets decreases.

[0069] A method for using a data mining system based on big data comprises the following steps:

[0070] S1. Connecting to the big data database: connecting the data mining system to the big data database so that the system can mine big data;

[0071] S2. Data processing: mining the big data database through the data preprocessing system, and cleaning, integrating, converting and simplifying the mined image data;

[0072] S3, Mining Execution: Classify the preprocessed image data, discover frequent patterns, correlations or causal relationships between data items, and analyze the trends and patterns of data changes over time;

[0073] S4, Data evaluation: Evaluate the mined image data to determine whether it is meaningful and reliable, and use visualization technology to display the mining results to users in the form of charts, graphs, etc., so that users can understand and interpret them, and generate interactive interfaces for query;

[0074] S5. Mining data processing: protecting the mined image data, evaluating the security of the mined data, and protecting the mined data;

[0075] S6. Connect to the enterprise database: transfer the image data from the mining data processing system to the enterprise database, perform collision training with the image database inside the enterprise database, and extract the familiar image data to avoid too much familiar image data, which will cause the image database to be bloated.

[0076] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by ordinary technicians in this technical field within the essential scope of the present invention also fall within the protection scope of the present invention.

Claims

1. A data mining system based on big data, comprising a big data database, a data mining system and a mining data processing system, characterized in that: Big data database: a large amount of unstructured or structured data from various sources; Data mining system: a system that mines specific image data from a large amount of data stored in a database, data warehouse or other information repository. The data mining system pre-processes the image data, performs mining and data evaluation; Mining data processing system: protect the mined image data, evaluate the security of the mined data, and protect the mined image data, including data representation and storage, data security control and data privacy protection.

2. The data mining system based on big data according to claim 1, characterized in that: The data preprocessing system includes data cleaning, data integration, data conversion and data reduction; Data cleaning: used to remove errors, duplications, inconsistencies and noise in corrected data; Data integration: combining data from different data sources into a consistent data set; Data conversion: normalize, standardize, discretize or encode data to meet the needs of mining algorithms; Data reduction: Reduce the amount of data through clustering, sampling and other techniques while trying to preserve the integrity of the original data.

3. The data mining system based on big data according to claim 1, characterized in that: The data mining algorithm execution system includes data classification, data clustering, association rule mining, anomaly detection and evolution analysis; Data classification: classify a dataset into predefined categories and predict the category of new data points; Data clustering: Divide a data set into multiple groups or clusters so that the data points within the same cluster are more similar, while the data points between different clusters are less similar; Association rule mining: discovering frequent patterns, correlations, or causal relationships between data items; Anomaly detection: Identifying data points that are significantly different from the majority of the data, i.e. outliers or outliers; Evolutionary analysis: Analyze trends and patterns in data over time.

4. The data mining system based on big data according to claim 1, characterized in that: The data evaluation system includes a data evaluation and generation interaction interface; Data evaluation: Evaluate the mined image data to determine whether it is meaningful and reliable, and use visualization technology to present the mining results to users in the form of charts, graphs, etc. to facilitate user understanding and interpretation; Generate interactive interface: Provide a user interactive interface that allows users to adjust mining parameters, select mining algorithms, or view detailed information about the mining process according to their needs.

5. The data mining system based on big data according to claim 1, characterized in that: The mining data processing system includes a data representation and storage module and a data security and privacy protection module; Data representation and storage module: represents the mined image data in an appropriate form and stores the image data in a database for subsequent query, analysis and application; Data security and privacy protection module: used to ensure the security and privacy of image data during data mining, prevent image data leakage or illegal use, and provide security measures such as data encryption and access control to protect user data security.

6. The data mining system based on big data according to claim 1, characterized in that: When the data mining algorithm performs data classification in the system, the calculation formula for judging the accuracy of data classification is: In the formula, G A (D) is the similarity between any two subsets in the data classification set; D is the total set of data classification sets; D1 and D2 are both subsets of the data classification set; P(D) is the prior probability of the data classification set of category D; P(x i / D) is the conditional probability of feature i under the condition of category D; P(xi) is the joint probability of i in all features D; The data obtained from the formula takes values ​​from 0 to 1: 0 means exactly the same; 1 means completely different. When the formula structure tends to 0, the similarity between the two subsets increases. When the formula structure tends to 1, the similarity between the two subsets decreases.

7. A method for using a data mining system based on big data, comprising a data mining system based on big data as claimed in any one of claims 1 to 6, characterized in that: The following steps are involved: S1. Connecting to the big data database: connecting the data mining system to the big data database so that the system can mine big data; S2. Data processing: mining the big data database through the data preprocessing system, and cleaning, integrating, converting and simplifying the mined image data; S3, Mining Execution: Classify the preprocessed image data, discover frequent patterns, correlations or causal relationships between data items, and analyze the trends and patterns of data changes over time; S4, Data evaluation: Evaluate the mined image data to determine whether it is meaningful and reliable, and use visualization technology to display the mining results to users in the form of charts, graphs, etc., so that users can understand and interpret them, and generate interactive interfaces for query; S5. Mining data processing: protecting the mined image data, evaluating the security of the mined data, and protecting the mined data; S6. Connect to the enterprise database: transfer the image data from the mining data processing system to the enterprise database, perform collision training with the image database inside the enterprise database, and extract the familiar image data to avoid too much familiar image data, which will cause the image database to be bloated.

Images