A method and system for mining Internet information data

Abstract

The present invention provides an Internet information data mining method and system, which relates to the field of information mining technology, including: collecting Internet information data and performing quality assessment and screening to obtain available data; performing data preprocessing on the available data to obtain target data; matching the target data with a target data mining algorithm to perform data mining to generate data mining results; and parsing and visualizing the data mining results. By performing data preprocessing on the available data obtained by performing quality assessment and screening on the collected Internet information data to obtain target data; matching the target data with a target data mining algorithm based on current business needs to perform data mining, and after obtaining data mining results, parsing and visualizing the data mining results, effective data mining of Internet information data can be achieved, which improves the quality and availability of data while also improving the scientificity and efficiency of enterprise decision-making.

Description

Technical Field

[0001] The present invention relates to the technical field of information mining, and in particular to an Internet information data mining method and system. Background Art

[0002] In recent years, with the rapid development of information technology, the amount of data on the Internet has exploded. Commercial data warehouses, social media, e-commerce platforms, etc. have accumulated massive amounts of data, providing rich resources for data mining. However, while a large amount of information brings convenience, it also frequently leads to the inability of the adopted data mining algorithms to match the current business scenarios due to the large amount of data, multiple types, information redundancy, difficulty in distinguishing true from false, and different forms. As a result, it is impossible to extract valuable data from massive information to provide support for corporate decision-making. Therefore, how to flexibly select and adjust data mining algorithms to achieve effective data mining of Internet information data in order to improve the scientific nature of corporate decision-making has become one of the current research focuses.

[0003] Therefore, the present invention provides an Internet information data mining method and system. Summary of the invention

[0004] The present invention provides an Internet information data mining method and system, which are used to obtain target data by performing data preprocessing on available data obtained by quality assessment and screening of collected Internet information data; matching target data mining algorithms to target data based on current business needs to perform data mining, and after obtaining data mining results, the data mining results are analyzed and visualized, thereby realizing effective data mining of Internet information data, improving the quality and availability of data, and also improving the scientificity and efficiency of enterprise decision-making.

[0005] The present invention provides an Internet information data mining method, comprising:

[0006] Step 1: Collect Internet information data and conduct quality assessment and screening to obtain usable data;

[0007] Step 2: Preprocess the available data to obtain target data;

[0008] Step 3: Perform data mining on the target data matching the target data mining algorithm to generate data mining results;

[0009] Step 4: Analyze and visualize the data mining results;

[0010] Step 3 includes:

[0011] Determine the application scenarios of current data mining results based on current business needs;

[0012] Extract the first algorithm matching the current result application scenario from the set scenario-algorithm mapping table, and set up a first algorithm list; if there is only a single first algorithm in the first algorithm list, regard the current first algorithm as the target data mining algorithm; if there are multiple first algorithms in the first algorithm list, calculate the mining matching degree of each first algorithm; regard the first algorithm with the highest mining matching degree as the target data mining algorithm;

[0013] Using the target data mining algorithm, model training is performed based on pre-acquired training data to obtain an initial mining model;

[0014] Inputting the target data into the data mining model obtained by testing and optimizing the initial mining model to generate data mining results;

[0015] The calculation formula of mining matching degree is as follows:

[0016] ; In the formula, It is expressed as the mining matching degree of the i-th first algorithm; It is represented as the specific applicable evaluation coefficient of the i-th first algorithm and the current result application scenario; It is expressed as the influence weight of the specific applicable evaluation on the mining matching degree of the determination algorithm; It is expressed as the weight of the impact of the quality of historical mining results on the mining matching degree of the determination algorithm; It is represented as the historical mining quality evaluation coefficient of the i-th first algorithm within the preset time period T; Represents the characteristic value of the jth data characteristic of the current target data, where j=1, 2, , n; n represents the total number of data characteristics; data characteristics include: data scale, data type, data structure and data distribution, among which the characteristic value of data scale refers to the storage space occupied by data, the characteristic value of data type refers to the range of values ​​that data can represent, the characteristic value of data structure refers to the access efficiency of data; the characteristic value of data distribution refers to the numerical value that quantifies the degree of dispersion of data; It is expressed as the influence weight of the j-th data feature on the calculation of the mining matching degree between the first algorithm and the current result application scenario.

[0017] Preferably, Internet information data is collected and quality assessed and screened to obtain usable data, including:

[0018] Determine the target data source based on current business needs;

[0019] Using a preset data capture technology, capture and aggregate data from the target data source to obtain a first data set;

[0020] Performing a comprehensive quality assessment on all data in the first data set based on a set data quality assessment indicator to obtain a first assessment coefficient;

[0021] Data with a first evaluation coefficient greater than a set quality evaluation threshold value is output as usable data from the first data set.

[0022] Preferably, the data quality assessment indicators are set to include data integrity, data accuracy, data timeliness and data consistency.

[0023] Preferably, the target data is input into a data mining model obtained by testing and optimizing the initial mining model to generate data mining results, including:

[0024] Performing data mining test on the initial mining model using pre-acquired test data to obtain data mining test results;

[0025] Tuning the parameters of the initial mining model based on the data mining test results to obtain tuned parameters;

[0026] Optimizing the initial mining model based on the tuned parameters to obtain a data mining model;

[0027] Inputting the preset experimental data into the data mining model for retesting to obtain re-mining test results;

[0028] Performing mining performance evaluation on the re-mining test results to determine whether the performance of the data mining model meets the requirements;

[0029] When the mining performance of the data mining model meets the requirements, the target data is input into the data mining model to generate data mining results;

[0030] When the mining performance of the data mining model does not meet the requirements, the model parameters of the data mining model are iteratively tuned and applied to the data mining model until the model performance meets the requirements, and then the target data is input into the data mining model to generate data mining results.

[0031] Preferably, determining the historical mining quality evaluation coefficient includes:

[0032] Obtaining a first historical mining result obtained by performing data mining using the first algorithm in a current result application scenario within a preset time period T;

[0033] Obtain a second historical mining result obtained by performing data mining using the first algorithm in a similar scenario with an application direction consistent with the current result application scenario within a preset time period T;

[0034] For the first historical mining result, performing mining quality evaluation using a set performance evaluation mechanism to obtain a first quality evaluation value;

[0035] Using the first quality evaluation value to establish a first historical mining quality curve in time series;

[0036] Extracting trend features from the first historical mining quality curve and performing weighted averaging to obtain a first quality trend coefficient;

[0037] For the second historical mining result, performing mining quality evaluation using a set performance evaluation mechanism to obtain a second quality evaluation value;

[0038] Using the second quality evaluation value to establish a second historical mining quality curve for each similar scene in time sequence;

[0039] Extract trend features from the second historical mining quality curve and perform weighted averaging to obtain a second quality trend coefficient;

[0040] Extract historical data characteristics of historical business data of each similar scenario within a preset time period T;

[0041] Compare the historical data characteristics with the data characteristics of the current target data and establish a characteristic difference matrix;

[0042] Calculate the difference eigenvalues ​​of all feature difference matrices of the current similar scenes and conduct a comprehensive analysis to obtain the feature difference value;

[0043] Determining a historical mining quality evaluation coefficient based on the first quality evaluation value, the second quality evaluation value, the first quality trend coefficient, the second quality trend coefficient, and the characteristic difference value;

[0044] Among them, the calculation formula of the historical mining quality evaluation coefficient is as follows:

[0045] ; In the formula, It is represented as the quality evaluation coefficient of historical mining using the i-th first algorithm; It is represented as the average value of the first quality evaluation values ​​corresponding to all the first historical mining results of the i-th first algorithm; It is represented as the first quality trend coefficient of the i-th first algorithm applied in the current result application scenario history; It is represented as the average value of the second quality evaluation value corresponding to the second historical mining result obtained by using the i-th first algorithm to perform data mining in the k-th similar scenario, where k=1, 2, 3, , m; m represents the total number of similar scenes; It is represented by the characteristic difference value between the historical business data used in the kth similar scenario and the current target data; It is expressed as the scenario similarity coefficient between the kth similar scenario and the current result application scenario, and its value range is ; Expressed as a scene reference factor; It is represented as the frequency of applying the i-th first algorithm to the k-th similar scene within the preset time period T; It is represented as the second quality trend coefficient of the k-th similar scene history using the i-th first algorithm; is expressed as the natural logarithm; e is expressed as the base of the natural logarithm.

[0046] Preferably, tuning the parameters of the initial mining model based on the data mining test results to obtain tuned parameters includes:

[0047] Performing test evaluation on the data mining test result based on set mining evaluation indicators to obtain a test evaluation result for each set mining evaluation indicator;

[0048] Compare the test evaluation result with the corresponding set mining evaluation index threshold, and then regard the corresponding set mining evaluation index of the test evaluation result that is less than the set mining evaluation index threshold as the first mining evaluation index;

[0049] Combine the first mining evaluation index with the current business needs for analysis to determine the optimization goal;

[0050] Based on the optimization goal, determining the parameters that need to be tuned in the initial mining model;

[0051] According to the model category of the initial mining model and the parameters to be tuned, a matching parameter tuning algorithm is used to calculate the optimal solution of the parameters to be tuned based on the optimization target, and the optimal solution is output as the tuned parameters.

[0052] The present invention provides an Internet information data mining system, comprising:

[0053] Data collection module: used to collect Internet information data and conduct quality assessment and screening to obtain usable data;

[0054] Data processing module: used for performing data preprocessing on the available data to obtain target data;

[0055] Data mining module: used to perform data mining on the target data matching the target data mining algorithm and generate data mining results;

[0056] Display module: used to analyze and visualize the data mining results.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] The target data is obtained by preprocessing the available data obtained by quality assessment and screening of the collected Internet information data; data mining is carried out by matching the target data with the target data mining algorithm based on the current business needs, and after obtaining the data mining results, the data mining results are analyzed and visualized, which can realize effective data mining of Internet information data, improve the quality and availability of data, and also improve the scientificity and efficiency of enterprise decision-making.

[0059] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0060] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0062] Figure 1 This is a flow chart of an Internet information data mining method according to an embodiment of the present invention;

[0063] Figure 2 The structure diagram of an Internet information data mining system in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0065] The embodiment of the present invention provides an Internet information data mining method, such as Figure 1 As shown, including:

[0066] Step 1: Collect Internet information data and conduct quality assessment and screening to obtain usable data;

[0067] Step 2: Preprocess the available data to obtain target data;

[0068] Step 3: Perform data mining on the target data matching the target data mining algorithm to generate data mining results;

[0069] Step 4: Analyze and visualize the data mining results;

[0070] Step 3 includes:

[0071] Determine the application scenarios of current data mining results based on current business needs;

[0072] Extract the first algorithm matching the current result application scenario from the set scenario-algorithm mapping table, and set up a first algorithm list; if there is only a single first algorithm in the first algorithm list, regard the current first algorithm as the target data mining algorithm; if there are multiple first algorithms in the first algorithm list, calculate the mining matching degree of each first algorithm; regard the first algorithm with the highest mining matching degree as the target data mining algorithm;

[0073] Using the target data mining algorithm, model training is performed based on pre-acquired training data to obtain an initial mining model;

[0074] Inputting the target data into the data mining model obtained by testing and optimizing the initial mining model to generate data mining results;

[0075] The calculation formula of mining matching degree is as follows:

[0076] ; In the formula, It is expressed as the mining matching degree of the i-th first algorithm; It is represented as the specific applicable evaluation coefficient of the i-th first algorithm and the current result application scenario; It is expressed as the influence weight of the specific applicable evaluation on the mining matching degree of the determination algorithm; It is expressed as the weight of the impact of the quality of historical mining results on the mining matching degree of the determination algorithm; It is represented as the historical mining quality evaluation coefficient of the i-th first algorithm within the preset time period T; Represents the characteristic value of the jth data characteristic of the current target data, where j=1, 2, , n; n represents the total number of data characteristics; data characteristics include: data scale, data type, data structure and data distribution, among which the characteristic value of data scale refers to the storage space occupied by data, the characteristic value of data type refers to the range of values ​​that data can represent, the characteristic value of data structure refers to the access efficiency of data; the characteristic value of data distribution refers to the numerical value that quantifies the degree of dispersion of data; It is expressed as the influence weight of the j-th data feature on the calculation of the mining matching degree between the first algorithm and the current result application scenario.

[0077] In this embodiment, Internet information data refers to data screened out based on current business needs from various types of data generated, transmitted, stored and processed through the Internet, wherein Internet information data covers four aspects: user behavior, transaction records, social media interactions and search queries, including text and image forms; current business needs refer to predicting sales trends, identifying fraudulent transactions or transaction risk assessment; available data refers to data obtained by quality assessment and then screening of data captured from the target data source using a preset data capture technology, wherein quality assessment refers to evaluating data using set data quality assessment indicators, and the set data quality assessment indicators are pre-set and include data integrity, data accuracy, data timeliness and data consistency.

[0078] In this embodiment, the target data is the data obtained after normalizing and cleaning the available data; the target data mining algorithm is the data mining algorithm that matches the target data; the data mining algorithm includes clustering analysis, classification, association rule mining, time series analysis and other algorithms; the visual display is to intuitively display the data mining results through charts, graphs, etc., to provide strong support for making business decisions.

[0079] In this embodiment, business needs refer to the expectations raised by the current enterprise or organization during the operation process. The current business needs include three needs: predicting sales trends, identifying fraudulent transactions, and evaluating transaction risks. The result application scenario refers to the scenario for using data mining results determined based on the current business needs. For example, when the current business need is to predict sales trends, the corresponding result application scenario is sales forecasting.

[0080] In this embodiment, the setting scenario-algorithm mapping table is a mapping table established based on the result application scenario and the corresponding applicable data mining algorithm, including the scenario description, the mining algorithm and the algorithm characteristics; the first algorithm is a matching algorithm screened out from the setting scenario-algorithm mapping table according to the determined result application scenario; the first algorithm list is a list composed of a first algorithm set; the data mining algorithm includes algorithms such as cluster analysis, classification, association rule mining, and time series analysis.

[0081] In this embodiment, the mining matching degree is used to characterize the degree of adaptation of the first algorithm to the current result application scenario; the initial mining model is a data mining model obtained by using the target data mining algorithm to perform model training based on pre-acquired training data.

[0082] In this embodiment, the specific applicability evaluation coefficient is obtained by extracting the scores of all field experts on the adaptability of the first algorithm to the current result application scenario within a preset historical time period and calculating the average difference; the influence weight of the specific applicability evaluation on the mining matching degree of the determination algorithm It refers to the average value of the impact of a specific applicable evaluation coefficient on the accuracy of the algorithm matching scenario under the application scenario of the historical algorithm matching scenario; the weight of the impact of the quality of historical mining results on the mining matching degree of the algorithm It refers to the average value of the impact of the quality effect of historical data mining on the accuracy of algorithm matching scenarios under the application scenarios of historical algorithm matching scenarios; data characteristics include data scale, data type, data structure and data distribution.

[0083] In this embodiment, the characteristic value of the data size of the target data refers to the storage space occupied by the data; the characteristic value of the data type of the target data refers to the value range that the data can represent, which is to map the value range of the target data to Obtained; the characteristic value of the data structure of the target data refers to the access efficiency of the corresponding available data of the current target data without data preprocessing; the characteristic value of the data distribution of the target data refers to the value used to quantify the degree of dispersion of the corresponding available data of the current target data without data preprocessing, that is, the skewness.

[0084] The beneficial effects of the above technical solution are: by performing quality assessment and screening on the collected Internet information data, the available data is obtained through data preprocessing to obtain the target data; based on the current business needs, the target data is matched with the target data mining algorithm for data mining, and after obtaining the data mining results, the data mining results are analyzed and visualized, which can realize effective data mining of Internet information data, improve the quality and availability of data, and also improve the scientificity and efficiency of enterprise decision-making.

[0085] The embodiment of the present invention provides an Internet information data mining method, which collects Internet information data and performs quality assessment and screening to obtain usable data, including:

[0086] Determine the target data source based on current business needs;

[0087] Using a preset data capture technology, capture and aggregate data from the target data source to obtain a first data set;

[0088] Performing a comprehensive quality assessment on all data in the first data set based on a set data quality assessment indicator to obtain a first assessment coefficient;

[0089] Data with a first evaluation coefficient greater than a set quality evaluation threshold value is output as usable data from the first data set.

[0090] In this embodiment, the target data source is determined based on business needs, including social media platforms, news websites, e-commerce platforms, user comments, etc., wherein business needs refer to the expectations of the current enterprise or organization during operations, including predicting sales trends, identifying fraudulent transactions, and risk assessment; the preset data capture technology is pre-set, such as web crawlers and API interfaces; the first data set is a data set established by collecting data captured from the target data source using the preset data capture technology, for example, based on the demand for predicting sales trends, using web crawlers to capture data on social media platforms to obtain social media interaction data, and the collection is output as the first data set.

[0091] In this embodiment, the set data quality assessment indicators are pre-set, including data integrity, data accuracy, data timeliness and data consistency; the first evaluation coefficient is a coefficient obtained by weighted averaging the indicator evaluation values ​​obtained by quality assessment of the data in the first data set based on the set data quality assessment indicators, and is used to express the data quality and reliability in the first data set, wherein the weight assigned to the set data quality assessment indicator is a value obtained by obtaining the corresponding set data quality assessment indicator weights in historical application scenarios similar to the current business scenario within a preset time period and calculating the mean difference; the set quality assessment threshold is pre-set, and is a value obtained by obtaining the corresponding set quality assessment thresholds in historical application scenarios similar to the current business scenario within a preset time period and calculating the average; available data refers to data in the first data set whose first evaluation coefficient is greater than the set quality assessment threshold.

[0092] In this embodiment, for example, the quality evaluation index values ​​of data u based on data integrity, data accuracy, data timeliness and data consistency are 0.8, 0.95, 0.75 and 0.6 respectively; based on the weights of data integrity, data accuracy, data timeliness and data consistency being r1, r2, r3 and r4, the first evaluation coefficient of data u is determined to be equal to , where 4 refers to the total number of set data quality assessment indicators, which include data integrity, data accuracy, data timeliness, and data consistency.

[0093] In this embodiment, for example, the first evaluation coefficients of data d1, d2, and d3 in the first data set d are c1, c2, and c3 respectively, where c1 is greater than the set quality evaluation threshold, and c2 and c3 are both less than the set quality evaluation threshold. At this time, the data d1 in the first data set d is output as available data.

[0094] The beneficial effects of the above technical solution are: determining the data source by analyzing business needs and performing data capture to generate a first data set; performing data quality assessment and screening on the first data set to obtain usable data, which can provide high-quality data support for subsequent data mining.

[0095] The embodiment of the present invention provides an Internet information data mining method, which inputs target data into a data mining model obtained by testing and optimizing the initial mining model to generate data mining results, including:

[0096] Performing data mining test on the initial mining model using pre-acquired test data to obtain data mining test results;

[0097] Tuning the parameters of the initial mining model based on the data mining test results to obtain tuned parameters;

[0098] Optimizing the initial mining model based on the tuned parameters to obtain a data mining model;

[0099] Inputting the preset experimental data into the data mining model for retesting to obtain re-mining test results;

[0100] Performing mining performance evaluation on the re-mining test results to determine whether the performance of the data mining model meets the requirements;

[0101] When the mining performance of the data mining model meets the requirements, the target data is input into the data mining model to generate data mining results;

[0102] When the mining performance of the data mining model does not meet the requirements, the model parameters of the data mining model are iteratively tuned and applied to the data mining model until the model performance meets the requirements, and then the target data is input into the data mining model to generate data mining results.

[0103] In this embodiment, the data mining test result refers to the test result obtained by testing the initial mining model using pre-acquired test data; the re-mining test result is the data mining result output by inputting preset experimental data into the data mining model for data mining; judging whether the performance of the data mining model meets the requirements means that when the quality evaluation value determined by evaluating the re-test result based on the set performance evaluation mechanism is greater than the quality evaluation value set in advance based on the current business needs, it is determined that the current data mining model meets the requirements, otherwise it is determined that the current data mining model does not meet the requirements.

[0104] In this embodiment, the tuned parameters are obtained by optimizing the adjustable parameters in the initial mining model; the data mining model is a model obtained by optimizing the initial mining model using the tuned parameters that optimize the parameters of the initial mining model based on the test results; the re-mining test results refer to the test results obtained by retesting the data mining model using preset experimental data, and the preset experimental data are pre-acquired data.

[0105] The beneficial effects of the above technical solution are: through scientific screening of the analysis of business needs, the target data mining algorithm is obtained, which effectively improves the accuracy and efficiency of data mining; the data mining model obtained after tuning the model built using the target data mining algorithm is used to perform data mining on the acquired target data to generate data mining results, which can provide strong and reliable data support for the company's decision-making.

[0106] The embodiment of the present invention provides an Internet information data mining method, which determines a historical mining quality evaluation coefficient, including:

[0107] Obtaining a first historical mining result obtained by performing data mining using the first algorithm in a current result application scenario within a preset time period T;

[0108] Obtain a second historical mining result obtained by performing data mining using the first algorithm in a similar scenario with an application direction consistent with the current result application scenario within a preset time period T;

[0109] For the first historical mining result, performing mining quality evaluation using a set performance evaluation mechanism to obtain a first quality evaluation value;

[0110] Using the first quality evaluation value to establish a first historical mining quality curve in time series;

[0111] Extracting trend features from the first historical mining quality curve and performing weighted averaging to obtain a first quality trend coefficient;

[0112] For the second historical mining result, performing mining quality evaluation using a set performance evaluation mechanism to obtain a second quality evaluation value;

[0113] Using the second quality evaluation value to establish a second historical mining quality curve for each similar scene in time sequence;

[0114] Extract trend features from the second historical mining quality curve and perform weighted averaging to obtain a second quality trend coefficient;

[0115] Extract historical data characteristics of historical business data of each similar scenario within a preset time period T;

[0116] Compare the historical data characteristics with the data characteristics of the current target data and establish a characteristic difference matrix;

[0117] Calculate the difference eigenvalues ​​of all feature difference matrices of the current similar scenes and conduct a comprehensive analysis to obtain the feature difference value;

[0118] Determining a historical mining quality evaluation coefficient based on the first quality evaluation value, the second quality evaluation value, the first quality trend coefficient, the second quality trend coefficient, and the characteristic difference value;

[0119] Among them, the calculation formula of the historical mining quality evaluation coefficient is as follows:

[0120] ; In the formula, It is represented as the quality evaluation coefficient of historical mining using the i-th first algorithm; It is represented as the average value of the first quality evaluation values ​​corresponding to all the first historical mining results of the i-th first algorithm; It is represented as the first quality trend coefficient of the i-th first algorithm applied in the current result application scenario history; It is represented as the average value of the second quality evaluation value corresponding to the second historical mining result obtained by using the i-th first algorithm to perform data mining in the k-th similar scenario, where k=1, 2, 3, , m; m represents the total number of similar scenes; It is represented by the characteristic difference value between the historical business data used in the kth similar scenario and the current target data; It is expressed as the scenario similarity coefficient between the kth similar scenario and the current result application scenario, and its value range is ; Expressed as a scene reference factor; It is represented as the frequency of applying the i-th first algorithm to the k-th similar scene within the preset time period T; It is represented as the second quality trend coefficient of the k-th similar scene history using the i-th first algorithm; is expressed as the natural logarithm; e is expressed as the base of the natural logarithm.

[0121] In this embodiment, the preset time period T is pre-set; the first historical mining result refers to the historical mining result generated after data mining the corresponding historical business data using the first algorithm in the current result application scenario within the preset time period T; application directions include prediction, classification, association, clustering, etc.

[0122] In this embodiment, the second historical mining result refers to the historical mining result generated after data mining the corresponding historical business data using the first algorithm in a similar scenario consistent with the application direction of the current result application scenario within a preset time period T; setting the performance evaluation mechanism is pre-set, which means performing data mining quality evaluation on the historical mining results based on the set mining evaluation indicators, and then averaging the indicator values ​​obtained by the evaluation to obtain the corresponding quality evaluation value; setting the mining evaluation indicators is pre-set based on business needs, including accuracy, precision, recall rate, F1 score, mean square error, etc., wherein business needs refer to the expectations put forward by the current enterprise or organization during the operation process, such as predicting sales trends, identifying fraudulent transactions, and risk assessment, etc.

[0123] In this embodiment, the first quality evaluation value is obtained by performing a mining quality evaluation on the first historical mining result based on a set performance evaluation mechanism, and is used to quantify the historical data mining quality of the current first algorithm in the current result application scenario; the first historical mining quality curve is a curve constructed in time series with the first quality evaluation value as the dependent variable; the first quality trend coefficient is obtained by weighted averaging the trend features extracted from the first historical mining quality curve, and is used to characterize the degree of change in the historical mining quality of data mining using the first algorithm in the current result application scenario within a preset time period T, wherein the trend features include the slope of the curve, the sum of the deviations of the extreme points from the average value, the curvature, and the acceleration; the weights assigned to the trend features are obtained by solving a matrix constructed by using the hierarchical analysis method to compare the trend features pairwise and score their relative importance.

[0124] In this embodiment, the second quality evaluation value is obtained by performing a mining quality evaluation on the second historical mining result based on a set performance evaluation mechanism, and is used to quantify the historical data mining quality of the current first algorithm in a similar scenario with an application direction consistent with the current result application scenario; the second historical mining quality curve is a curve constructed in time series with the second quality evaluation value as the dependent variable; the second quality trend coefficient is obtained by weighted averaging the trend features extracted from the second historical mining quality curve, and is used to characterize the degree of change in the historical mining quality of data mining using the first algorithm in a similar scenario with an application direction consistent with the current result application scenario within a preset time period T, wherein the trend features include the slope of the curve, the sum of the deviations of the extreme points from the average value, the curvature, and the acceleration.

[0125] In this embodiment, data characteristics include data scale, data type, data structure and data distribution; the characteristic difference matrix is ​​a matrix established by comparing the historical data characteristics of historical business data obtained at different data mining moments in similar scenarios with the data characteristics of current target data, wherein the target data refers to data after data preprocessing of available data; historical business data refers to historical data used for data mining in similar scenarios; the characteristic difference value is obtained by summarizing and analyzing the difference eigenvalues ​​of all characteristic difference matrices of the current similar scenarios, and is used to express the degree of difference in data characteristics between the historical business data used for data mining and the current target data in similar scenarios.

[0126] In this embodiment, for example, there are historical business data p1, p2, and p3 similar to scenario 1. The historical data characteristics of the historical business data p1, p2, and p3 are respectively compared with the data characteristics of the current target data and a matrix is ​​established to obtain feature difference matrices X1, X2, and X3;

[0127] Calculate the feature difference matrices X1, X2, and X3 to obtain the eigenvalues ​​in turn , , And as the difference feature value, at this time, the characteristic difference value between the historical business data used for data mining in similar scenario 1 and the current target data .

[0128] In this embodiment, the historical mining quality evaluation coefficient is used to express the historical data mining quality of the first algorithm in the current result application scenario and similar scenarios with the same application direction as the current result application scenario; the scenario similarity coefficient is obtained by evaluating the similarity between similar scenarios and the current result application scenario based on set similarity indicators, wherein the set similarity indicators include the business objectives of the scenario, the expected results, the complexity of the problems to be solved, and the technical dependence of the current algorithm on the technical environment.

[0129] The beneficial effect of the above technical solution is: by analyzing the historical data mining quality of the current algorithm in the current result application scenario and similar scenarios with the same application direction as the current result application scenario, the historical mining quality evaluation coefficient can be obtained, which provides a data basis for calculating the mining matching degree, which is conducive to the scientific selection of the target data mining algorithm that best matches the current result application scenario, and improves the accuracy and efficiency of subsequent data mining.

[0130] The embodiment of the present invention provides an Internet information data mining method, which optimizes the parameters of the initial mining model based on the data mining test results to obtain the optimized parameters, including:

[0131] Performing test evaluation on the data mining test result based on set mining evaluation indicators to obtain a test evaluation result for each set mining evaluation indicator;

[0132] Compare the test evaluation result with the corresponding set mining evaluation index threshold, and then regard the corresponding set mining evaluation index of the test evaluation result that is less than the set mining evaluation index threshold as the first mining evaluation index;

[0133] Combine the first mining evaluation index with the current business needs for analysis to determine the optimization goal;

[0134] Based on the optimization goal, determining the parameters that need to be tuned in the initial mining model;

[0135] According to the model category of the initial mining model and the parameters to be tuned, a matching parameter tuning algorithm is used to calculate the optimal solution of the parameters to be tuned based on the optimization target, and the optimal solution is output as the tuned parameters.

[0136] In this embodiment, the data mining test result refers to the test result obtained by testing the initial mining model using pre-acquired test data, wherein the initial mining model is a data mining model obtained by using a target data mining algorithm to perform model training based on pre-acquired training data.

[0137] In this embodiment, the mining evaluation indicators are set in advance based on business needs, including accuracy, precision, recall rate, F1 score, mean square error, etc., where business needs refer to the expectations put forward by the current enterprise or organization during the operation process, including predicting sales trends, identifying fraudulent transactions, and risk assessment.

[0138] In this embodiment, the test evaluation result refers to the indicator evaluation value obtained by testing and evaluating the data mining test result based on the set mining evaluation indicator, which is used to characterize the data mining quality of the initial mining model; the set mining evaluation indicator threshold is pre-set, and is the peak value obtained from the curve established by the historical data mining evaluation values ​​obtained in the historical data mining scenario within a preset time period; the first mining evaluation indicator refers to the set mining evaluation indicator that is less than the set mining evaluation indicator threshold.

[0139] In this embodiment, the optimization goal is determined based on business needs. For example, if the business need is to identify fraudulent transactions, the corresponding optimization goal is to maximize accuracy. The parameters to be tuned are parameters selected according to the optimization goal from the parameter set determined based on the category of the initial mining model, such as hyperparameters. Different goals may require adjustment of different parameters, and different categories of models have different parameter sets. Model categories include decision trees, random forests, neural networks, etc. Parameter tuning algorithms include random search, Bayesian optimization, etc., which are used to find the optimal solution for the parameters to be tuned. Matching the appropriate parameter tuning algorithm means selecting a parameter tuning algorithm based on the initial mining model from all parameter tuning algorithms. The algorithm with the largest matching value is determined by comprehensive analysis of the model category of the mining model and the characteristics of the parameters that need to be tuned, where the matching value refers to the value obtained by weighted averaging the sensitive value of the parameter tuning algorithm for the initial mining model category and the applicable value of the characteristics of the parameters that need to be tuned, where the sensitive value refers to the frequency of parameter optimization of the model with the same category as the initial mining model using the parameter tuning algorithm within a preset time period; the applicable value is obtained by comprehensive analysis of the historical tuning effects of the parameter tuning algorithm on the parameters that need to be tuned with different characteristics of the parameters that need to be tuned; the characteristics of the parameters that need to be tuned include the interaction between the parameters that need to be tuned and the number of parameters that need to be tuned.

[0140] The beneficial effect of the above technical solution is: by tuning the parameters of the initial mining model based on the data mining test results, the tuned parameters can be obtained, which can provide data support for model optimization, thereby improving the model data mining performance and providing a more reliable basis for the company's business decisions.

[0141] The embodiment of the present invention provides an Internet information data mining system, such as Figure 2 As shown, including:

[0142] Data collection module: used to collect Internet information data and conduct quality assessment and screening to obtain usable data;

[0143] Data processing module: used for performing data preprocessing on the available data to obtain target data;

[0144] Data mining module: used to perform data mining on the target data matching the target data mining algorithm and generate data mining results;

[0145] Display module: used to analyze and visualize the data mining results.

[0146] The beneficial effects of the above technical solution are: by performing quality assessment and screening on the collected Internet information data, the available data is obtained through data preprocessing to obtain the target data; based on the current business needs, the target data is matched with the target data mining algorithm for data mining, and after obtaining the data mining results, the data mining results are analyzed and visualized, which can realize effective data mining of Internet information data, improve the quality and availability of data, and also improve the scientificity and efficiency of enterprise decision-making.

[0147] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for mining Internet information data, characterized in that: include: Step 1: Collect Internet information data and conduct quality assessment and screening to obtain usable data; Step 2: Preprocess the available data to obtain target data; Step 3: Perform data mining on the target data matching the target data mining algorithm to generate data mining results; Step 4: Analyze and visualize the data mining results; Step 3 includes: Determine the application scenarios of current data mining results based on current business needs; Extract the first algorithm matching the current result application scenario from the set scenario-algorithm mapping table, and set up a first algorithm list; if there is only a single first algorithm in the first algorithm list, regard the current first algorithm as the target data mining algorithm; if there are multiple first algorithms in the first algorithm list, calculate the mining matching degree of each first algorithm; regard the first algorithm with the highest mining matching degree as the target data mining algorithm; Using the target data mining algorithm, model training is performed based on pre-acquired training data to obtain an initial mining model; Inputting the target data into the data mining model obtained by testing and optimizing the initial mining model to generate data mining results; The calculation formula of mining matching degree is as follows: ; In the formula, It is expressed as the mining matching degree of the i-th first algorithm; It is represented as the specific applicable evaluation coefficient of the i-th first algorithm and the current result application scenario; It is expressed as the influence weight of the specific applicable evaluation on the mining matching degree of the determination algorithm; It is expressed as the weight of the impact of the quality of historical mining results on the mining matching degree of the determination algorithm; It is represented as the historical mining quality evaluation coefficient of the i-th first algorithm within the preset time period T; Represents the characteristic value of the jth data characteristic of the current target data, where j=1, 2, , n; n represents the total number of data characteristics; data characteristics include: data scale, data type, data structure and data distribution, among which the characteristic value of data scale refers to the storage space occupied by data, the characteristic value of data type refers to the range of values ​​that data can represent, the characteristic value of data structure refers to the access efficiency of data; the characteristic value of data distribution refers to the numerical value that quantifies the degree of dispersion of data; It is expressed as the influence weight of the j-th data feature on the calculation of the mining matching degree between the first algorithm and the current result application scenario.

2. The method for mining Internet information data according to claim 1, characterized in that: Collect Internet information data and conduct quality assessment and screening to obtain usable data, including: Determine the target data source based on current business needs; Using a preset data capture technology, capture and aggregate data from the target data source to obtain a first data set; Performing a comprehensive quality assessment on all data in the first data set based on a set data quality assessment indicator to obtain a first assessment coefficient; Data with a first evaluation coefficient greater than a set quality evaluation threshold value is output as usable data from the first data set.

3. The method for Internet information data mining according to claim 2, characterized in that: The data quality assessment indicators set include data completeness, data accuracy, data timeliness and data consistency.

4. The method for mining Internet information data according to claim 1, characterized in that: Inputting the target data into the data mining model obtained by testing and optimizing the initial mining model to generate data mining results, including: Using the target data mining algorithm, model training is performed based on pre-acquired training data to obtain an initial mining model; Performing data mining test on the initial mining model using pre-acquired test data to obtain data mining test results; Tuning the parameters of the initial mining model based on the data mining test results to obtain tuned parameters; Optimizing the initial mining model based on the tuned parameters to obtain a data mining model; Inputting the preset experimental data into the data mining model for retesting to obtain re-mining test results; Performing mining performance evaluation on the re-mining test results to determine whether the performance of the data mining model meets the requirements; When the mining performance of the data mining model meets the requirements, the target data is input into the data mining model to generate data mining results; When the mining performance of the data mining model does not meet the requirements, the model parameters of the data mining model are iteratively tuned and applied to the data mining model until the model performance meets the requirements, and then the target data is input into the data mining model to generate data mining results.

5. The method for mining Internet information data according to claim 1, characterized in that: Determine the historical mining quality evaluation coefficient, including: Obtaining a first historical mining result obtained by performing data mining using the first algorithm in a current result application scenario within a preset time period T; Obtain a second historical mining result obtained by performing data mining using the first algorithm in a similar scenario with an application direction consistent with the current result application scenario within a preset time period T; For the first historical mining result, performing mining quality evaluation using a set performance evaluation mechanism to obtain a first quality evaluation value; Using the first quality evaluation value to establish a first historical mining quality curve in time series; Extracting trend features from the first historical mining quality curve and performing weighted averaging to obtain a first quality trend coefficient; For the second historical mining result, performing mining quality evaluation using a set performance evaluation mechanism to obtain a second quality evaluation value; Using the second quality evaluation value to establish a second historical mining quality curve for each similar scene in time sequence; Extract trend features from the second historical mining quality curve and perform weighted averaging to obtain a second quality trend coefficient; Extract historical data characteristics of historical business data of each similar scenario within a preset time period T; Compare the historical data characteristics with the data characteristics of the current target data and establish a characteristic difference matrix; Calculate the difference eigenvalues ​​of all feature difference matrices of the current similar scenes and conduct a comprehensive analysis to obtain the feature difference value; Determining a historical mining quality evaluation coefficient based on the first quality evaluation value, the second quality evaluation value, the first quality trend coefficient, the second quality trend coefficient, and the characteristic difference value; Among them, the calculation formula of the historical mining quality evaluation coefficient is as follows: ; In the formula, It is represented as the quality evaluation coefficient of historical mining using the i-th first algorithm; It is represented as the average value of the first quality evaluation values ​​corresponding to all the first historical mining results of the i-th first algorithm; It is represented as the first quality trend coefficient of the i-th first algorithm applied in the current result application scenario history; It is represented as the average value of the second quality evaluation value corresponding to the second historical mining result obtained by using the i-th first algorithm to perform data mining in the k-th similar scenario, where k=1, 2, 3, , m; m represents the total number of similar scenes; It is represented by the characteristic difference value between the historical business data used in the kth similar scenario and the current target data; It is expressed as the scenario similarity coefficient between the kth similar scenario and the current result application scenario, and its value range is ; Expressed as a scene reference factor; It is represented as the frequency of applying the i-th first algorithm to the k-th similar scene within the preset time period T; It is represented as the second quality trend coefficient of the k-th similar scene history using the i-th first algorithm; is expressed as the natural logarithm; e is expressed as the base of the natural logarithm.

6. The method for mining Internet information data according to claim 4, characterized in that: Optimizing the parameters of the initial mining model based on the data mining test results to obtain optimized parameters includes: Performing test evaluation on the data mining test result based on set mining evaluation indicators to obtain a test evaluation result for each set mining evaluation indicator; Compare the test evaluation result with the corresponding set mining evaluation index threshold, and then regard the corresponding set mining evaluation index of the test evaluation result that is less than the set mining evaluation index threshold as the first mining evaluation index; Combine the first mining evaluation index with the current business needs for analysis to determine the optimization goal; Based on the optimization goal, determining the parameters that need to be tuned in the initial mining model; According to the model category of the initial mining model and the parameters to be tuned, a matching parameter tuning algorithm is used to calculate the optimal solution of the parameters to be tuned based on the optimization target, and the optimal solution is output as the tuned parameters.

7. An Internet information data mining system, characterized in that: include: Data collection module: used to collect Internet information data and conduct quality assessment and screening to obtain usable data; Data processing module: used for performing data preprocessing on the available data to obtain target data; Data mining module: used to perform data mining on the target data matching the target data mining algorithm and generate data mining results; Display module: used to analyze and visualize the data mining results; Among them, the data mining module includes: Determine the application scenarios of current data mining results based on current business needs; Extract the first algorithm matching the current result application scenario from the set scenario-algorithm mapping table, and set up a first algorithm list; if there is only a single first algorithm in the first algorithm list, regard the current first algorithm as the target data mining algorithm; if there are multiple first algorithms in the first algorithm list, calculate the mining matching degree of each first algorithm; regard the first algorithm with the highest mining matching degree as the target data mining algorithm; Using the target data mining algorithm, model training is performed based on pre-acquired training data to obtain an initial mining model; Inputting the target data into the data mining model obtained by testing and optimizing the initial mining model to generate data mining results; The calculation formula of mining matching degree is as follows: ; In the formula, It is expressed as the mining matching degree of the i-th first algorithm; It is represented as the specific applicable evaluation coefficient of the i-th first algorithm and the current result application scenario; It is expressed as the influence weight of the specific applicable evaluation on the mining matching degree of the determination algorithm; It is expressed as the weight of the impact of the quality of historical mining results on the mining matching degree of the determination algorithm; It is represented as the historical mining quality evaluation coefficient of the i-th first algorithm within the preset time period T; Represents the characteristic value of the jth data characteristic of the current target data, where j=1, 2, , n; n represents the total number of data characteristics; data characteristics include: data scale, data type, data structure and data distribution, among which the characteristic value of data scale refers to the storage space occupied by data, the characteristic value of data type refers to the range of values ​​that data can represent, the characteristic value of data structure refers to the access efficiency of data; the characteristic value of data distribution refers to the numerical value that quantifies the degree of dispersion of data; It is expressed as the influence weight of the j-th data feature on the calculation of the mining matching degree between the first algorithm and the current result application scenario.

Images