Network data mining method and device, equipment and storage medium

Abstract

The invention relates to the technical field of data mining, and provides a network data mining method and device and a storage medium, and the method comprises the steps: collecting data from different data sources; preprocessing data collected from different data sources to obtain preprocessed data of the data; for different data types of the preprocessed data, extracting feature data of the preprocessed data by using a preset algorithm based on deep learning; based on a machine learning algorithm, performing data mining on the feature data of the preprocessed data to obtain a data mining result; and displaying the data mining result to the user in a visual mode. According to the technical scheme provided by the invention, the data with real value on the network can be quickly and accurately mined.

Description

Technical Field

[0001] The present invention relates to the technical field of data mining, and in particular to a network data mining method, device, equipment and computer storage medium. Background Art

[0002] In today's Internet era, the amount of data is exploding, and mining network data to obtain valuable information has become crucial. In the prior art, some network data mining methods usually use simple keyword matching or rule-based methods to screen and analyze data. However, the methods in the prior art have many shortcomings, such as low accuracy, poor adaptability, and low efficiency. Summary of the invention

[0003] The present application provides a network data mining method, device, equipment and computer storage medium, which can quickly and accurately mine data with real value on the network.

[0004] In one aspect, the present application provides a network data mining method, the method comprising:

[0005] Collect data from different data sources;

[0006] Preprocessing the data collected from different data sources to obtain preprocessed data;

[0007] For different data types of the preprocessed data, extract feature data of the preprocessed data using a preset algorithm based on deep learning;

[0008] Based on a machine learning algorithm, data mining is performed on the feature data of the preprocessed data to obtain a data mining result;

[0009] The data mining results are displayed to the user in a visual manner.

[0010] On the other hand, the present application provides a network data mining device, the device comprising:

[0011] A collection module, used to collect data from different data sources;

[0012] A preprocessing module, used for preprocessing the data collected from different data sources to obtain preprocessed data of the data;

[0013] An extraction module, for extracting feature data of the preprocessed data using a preset algorithm based on deep learning for different data types of the preprocessed data;

[0014] A mining module, used for performing data mining on the feature data of the preprocessed data based on a machine learning algorithm to obtain a data mining result;

[0015] The display module is used to display the data mining results to the user in a visual manner.

[0016] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the technical solution of the above-mentioned network data mining method when executing the computer program.

[0017] In a fourth aspect, the present application provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the technical solution of the above-mentioned network data mining method.

[0018] From the technical solution provided by the present application, it can be seen that after collecting data from different data sources, the data collected from different data sources are preprocessed to obtain preprocessed data; for different data types of the preprocessed data, a preset algorithm based on deep learning is used to extract feature data of the preprocessed data; based on the machine learning algorithm, data mining is performed on the feature data of the preprocessed data to obtain data mining results. Since the preset algorithm based on deep learning is suitable for data of various types or patterns, it can accurately extract data features, and the machine learning algorithm shows excellent performance in data classification, and these algorithms are all executed by computer programs, even for large-scale data, the task can be executed in real time, that is, it has strong real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 is a flow chart of a network data mining method provided by an embodiment of the present application;

[0021] Figure 2 is a schematic diagram of the structure of a network data mining device provided in an embodiment of the present application;

[0022] Figure 3 It is a schematic diagram of the structure of the device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0025] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0026] In today's Internet era, the amount of data is exploding, and it is crucial to mine network data to obtain valuable information. In the prior art, some network data mining methods usually use simple keyword matching or rule-based methods to screen and analyze data. However, these network data mining methods also have many defects such as low accuracy, poor adaptability and low efficiency. For example, simple keyword matching is prone to produce a large number of false positives and false negatives, making it difficult to accurately mine truly valuable information; for another example, rule-based methods require a large number of rules to be set manually, and when the data pattern changes, these rules need to be readjusted, and the adaptability is poor; for another example, the existing methods are slow when processing large-scale data and cannot meet real-time requirements, etc.

[0027] In response to the above problems, the embodiments of the present application provide a network data mining method, apparatus, device and computer storage medium, which can quickly and accurately mine data with real value on the network.

[0028] The technical solution of the embodiment of the present application is described in detail below with reference to the accompanying drawings

[0029] In this specification, for the convenience of description, the sizes of various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0030] As attached Figure 1 As shown, this application proposes a flow chart of a network data mining method, which mainly includes steps S101 to S105, which are described in detail as follows:

[0031] Step S101: Collect data from different data sources.

[0032] It is understandable that the data source may be social media, news websites, forums, etc. In the embodiment of the present application, various types of data are collected from data sources such as social media, news websites, forums, etc. through web crawlers, API calls, and data streams.

[0033] Step S102: pre-processing the data collected from different data sources to obtain pre-processed data.

[0034] Since network data often comes from a wide range of sources and is messy, it may contain various noises and outliers. Directly processing this data may affect the analysis results; on the other hand, preprocessing the data can reduce the cost of data processing and storage and optimize computing resources. In other words, data preprocessing is the key to ensuring data quality, improving analysis results, and optimizing model performance. It lays a solid foundation for subsequent data mining and analysis, and can significantly improve the accuracy and reliability of analysis results. Therefore, after collecting data from different data sources, the data collected from different data sources can be preprocessed to obtain preprocessed data. As an embodiment of the present application, preprocessing the data collected from different data sources to obtain preprocessed data can be achieved through steps S1021 to S1023, as described in detail as follows:

[0035] Step S1021: clean the data collected from different data sources to obtain cleaned data.

[0036] As mentioned above, network data often comes from a wide range of sources and is messy. In order to process the data efficiently in the future, the first step is to clean the data collected from different data sources, including identifying and correcting missing values ​​in the data, detecting and correcting outliers, and removing duplicate data, etc. The following describes the specific implementation solutions of the above operations one by one:

[0037] 1) Identify and correct missing values ​​in the data. Specifically, regular expressions can be used to match missing identifiers (such as "NA", "NULL") or empty values, combined with natural language processing (NLP) technology, such as detecting possible missing information in text data through context analysis and recognition; use appropriate filling methods to make corrections. For numerical data, the mean, median or regression model can be used for filling, and for text data, NLP technology can be used to fill missing content through context inference or use sequence-to-sequence models to fill in, generate complete sentences based on the context, and so on.

[0038] 2) Detect and correct outliers. Specifically, outliers can be detected using standard deviation-based outlier detection or by identifying unreasonable expressions in text. For outliers in numerical data, reasonable values ​​can be predicted by regression models for correction. For outliers in text data, reasonable alternative content can be inferred based on language models.

[0039] 3) Remove duplicate data. Specifically, for numerical data, regular expressions can be used to accurately match them. For text data, similarity analysis can be performed (for example, through vector representation) to identify duplicate records, thereby detecting duplicate data and deleting or merging them.

[0040] Step S1022: Use any one of the Gaussian filtering algorithm, the moving average method and the entropy filtering or a combination thereof to remove the noise of the cleaned data to obtain denoised data.

[0041] In an embodiment of the present application, the use of a Gaussian filter algorithm to remove noise from cleaned data may specifically be: traversing the cleaned data point by point, processing one data point and its neighborhood at a time; performing a convolution operation on the Gaussian kernel matrix and the cleaned data, wherein for the edge of the data, the processing method may be zero filling, mirror filling, or boundary extension, etc., and for each data point, multiplying the Gaussian kernel matrix of the Gaussian kernel matrix with the data points in its neighborhood; and replacing the value of the original data point with the calculated weighted sum. In the process of denoising using the Gaussian filter algorithm, the denoising effect can be observed to ensure that the key features of the data are retained while the noise is reduced, to check whether over-smoothing occurs, and to adjust the standard deviation of the Gaussian function and / or the size of the Gaussian kernel matrix if necessary. Through the above scheme, the Gaussian filter algorithm and smoothed data can be effectively applied to reduce noise while retaining important structural information.

[0042] In an embodiment of the present application, a general scheme for removing noise from cleaned data using a moving average method may be: for each data point, the data point and several adjacent data points are averaged to reduce short-term fluctuations. Furthermore, a weighted moving average method may be used, which assigns a weight to each data point and performs weighted averaging of the data points according to the weight, thereby better smoothing the data and retaining important trends. The specific process includes: selecting a preset weight function, including linear weights, exponential weighting, etc.; determining the size of the window of the weighted moving average filter; calculating the weight of each point in the window according to the selected weight function; sliding the window on the cleaned data, and calculating the weighted average for each data point, and for the data points at the edge of the window, a filling or extension strategy may be used. The weighted moving average method provides a more flexible and sophisticated denoising technique by applying different weights to each data point. Compared with the usual moving average method, the weighted moving average method can better smooth the data while retaining important trends and features in the data.

[0043] In the embodiment of the present application, the general scheme of using entropy filtering to remove noise from cleaned data can be: first calculate the entropy value of each feature or variable in the cleaned data, the larger the entropy value, the more information the feature carries; according to the size of the entropy value, remove those features that carry less information, or adjust them to reduce noise. For example, in image processing, entropy filtering can be used to remove unimportant parts of an image. The data after entropy filtering will appear more streamlined, reduce redundant information, and provide a more efficient basis for subsequent data normalization.

[0044] Step S1023: normalize the denoised data to obtain preprocessed data.

[0045] As mentioned above, in network data mining, data usually comes from different sources and formats. On the one hand, data from different sources are converted into a consistent format for easy integration and analysis; on the other hand, through data fusion technology, information from multiple data sources is combined to provide a comprehensive perspective. Therefore, after obtaining the denoised data, the denoised data can be normalized to obtain the pre-processed data of the data. In an embodiment of the present application, normalizing the denoised data can be a Min-Max normalization or Z-score standardization scheme, wherein the Min-Max normalization scheme can specifically be to use a normalization formula to linearly scale the denoised data to a specified range, such as [0, 1], that is, by mapping the minimum value of the denoised data to 0, the maximum value to 1, and other values ​​​​are scaled according to a linear ratio, and the Z-score standardization scheme can specifically be to use the Z-score formula to adjust the denoised data to a standard normal distribution with a mean of 0 and a standard deviation of 1, so that the data is normalized by calculating the difference between each data point and the mean and dividing it by the standard deviation.

[0046] Step S103: for different data types of the preprocessed data, extract feature data of the preprocessed data using a preset algorithm based on deep learning.

[0047] Since the algorithm based on deep learning is adaptable to various types or patterns of data and can accurately extract data features, in an embodiment of the present application, for different data types of the preprocessed data, a preset algorithm based on deep learning is used to extract feature data of the preprocessed data, including: for different data types of the preprocessed data, using an improved convolutional neural network to extract feature data of the preprocessed data; or, for different data types of the preprocessed data, using an autoencoder to extract feature data of the preprocessed data.

[0048] As an embodiment of the present application, for different data types of preprocessed data, extracting feature data of the preprocessed data using the improved convolutional neural network can be implemented through steps S1031 to S1033, which are described in detail as follows:

[0049] Step S1031 uses a specific activation function to generate the weight of each channel based on the attention mechanism and assigns it to the corresponding channel.

[0050] In an embodiment of the present application, the activation function may be a Swish activation function or a Mish activation function, wherein the Swish activation function is characterized in that the output is small when the input value is small, and the output is close to linear when the value is large. Compared with the Swish activation function, the Mish activation function is smoother and can provide better performance in most tasks. The weight of each channel is generated based on the attention mechanism and assigned to the corresponding channel, so that the improved convolutional neural network can assign higher weights to important parts or channels in the image, thereby improving the final feature extraction accuracy.

[0051] As an embodiment of the present application, using a specific activation function, generating a weight value for each channel based on the attention mechanism and assigning it to the corresponding channel can be implemented through steps S1 to S3, as described in detail as follows:

[0052] Step S1: Perform global average pooling on each channel to generate a channel-level global feature vector.

[0053] First, global average pooling is performed on each channel to compress the spatial dimension into a scalar and generate a channel-level global feature vector, which can be achieved by the following calculation formula:

[0054]

[0055] Among them, z c is the global average pooling result of the cth channel, F c (h, w) represents the feature map of the cth channel, H and W are the height and width of the feature map respectively. After global pooling, a global feature vector with a dimension of 1*1*C is obtained, where C is the number of channels.

[0056] Step S2: The attention weight of each channel is obtained by mapping the global feature vector at the channel level and activating the output of the fully connected layer using a specific activation function.

[0057] Specifically, firstly, the global feature vector z obtained by pooling in step S1 is c Input to a fully connected layer and map it to a low-dimensional space. Usually the reduction factor r is a common hyperparameter used to reduce the amount of calculation. This mapping process can be expressed as:

[0058] v=W 1 z+b 1

[0059] Among them, W 1 ∈R C / r×C is the weight matrix, b 1 is the bias term, and v is the mapped vector.

[0060] Secondly, use the activation function to perform a nonlinear transformation on the output of the fully connected layer. The Swish activation function or the Mish activation function can be used here:

[0061] Swish(x)=x*sigmoid(x)

[0062] Mish(x)=x*tanh(ln(1+e x ))

[0063] This step introduces nonlinear transformation and enhances the expressive power of the network.

[0064] In the third step, the activated features are restored to the dimension of the original number of channels through the second fully connected layer:

[0065] s=W 2 v+b 2

[0066] Among them, W 2 ∈R C / r×C is the weight matrix of the second fully connected layer, b 2 is the bias term.

[0067] Finally, in order to control the generated weights between [0, 1], the Sigmoid activation function is used:

[0068]

[0069] Among them, s c is the attention weight of the cth channel.

[0070] Step S3: Reapply the attention weights of each channel to each channel.

[0071] Specifically, the generated weight s c ∈R C is multiplied back to each channel of the input feature map:

[0072] F'=s*F

[0073] ccc

[0074] Thereby, each channel is weighted to improve the expressiveness of important channels.

[0075] As another embodiment of the present application, using a specific activation function, generating a weight for each channel based on the attention mechanism and assigning it to the corresponding channel can be implemented by steps S'1 to S'7, as described in detail as follows:

[0076] Step S'1: Perform global average pooling and global maximum pooling on each channel to generate a first global feature vector and a second global feature vector at the channel level, respectively.

[0077]

[0078] in, is the global average pooling result of the cth channel, that is, the first global eigenvector, is the global maximum pooling result of the cth channel, that is, the second global eigenvector, and They are all feature vectors of dimension 1*1*C, where C is the number of channels.

[0079] Step S'2: Input the first global feature vector and the second global feature vector into a shared fully connected layer respectively.

[0080]

[0081] Among them, W 1 ∈R C / r×C The weight matrix of the fully connected layer, b 1 is the bias term, and the parameters of this fully connected layer are shared.

[0082] Step S'3: Use a specific activation function to perform nonlinear activation on the first global feature vector and the second global feature vector respectively.

[0083] The Swish activation function or the Mish activation function can be used here:

[0084] Swish(x)=x*sigmoid(x)

[0085] Mish(x)=x*tanh(ln(1+e x )).

[0086] Nonlinear activation is performed on the first global eigenvector and the second global eigenvector respectively.

[0087] Step S'4: The two activated vectors are restored to the original number of channels through the second fully connected layer.

[0088] Similar to inputting the first global feature vector and the second global feature vector into the shared fully connected layer respectively:

[0089] s avg =W 2 v avg +b 2

[0090] s max =W 2 v max +b 2

[0091] Step S'5: The two sets of channel weights are fused by element-by-element addition to obtain the fused weight s.

[0092] That is, s = s avg +s max .

[0093] Step S'6: Normalize the fused channel weights through the Sigmoid function to obtain the attention s of each channel c .

[0094] Right now,

[0095] Step S'7: Set the attention weight s of each channel c Reapply to each channel.

[0096] Specifically, the generated weight s c ∈R C is multiplied back to each channel of the input feature map:

[0097] F′ c =s c *F c

[0098] Steps S'1 to S'7 can more accurately adjust the importance of channels by simultaneously considering the channel features of maximum pooling and average pooling.

[0099] Step S1032: Perform convolution operation on the channels that have been input with preprocessed data and assigned weights, either individually or in groups, to generate corresponding feature maps.

[0100] In the embodiment of the present application, the convolution operation is performed separately on the channels that have been input with preprocessed data and weights, and the corresponding feature map is generated specifically by: applying the first convolution kernel to each channel for deep convolution to generate the corresponding feature map. The convolution operation is performed on the channels that have been input with preprocessed data and weights, and the corresponding feature map is generated specifically by: dividing the channels of the input feature map into several groups, and performing the convolution operation independently on each group; applying the second convolution kernel to fuse the channel information after each deep convolution together to generate a new feature map.

[0101] Step S1033: Fuse or concatenate the feature maps of all channels to obtain feature data of the preprocessed data.

[0102] Specifically, if step S1032 is to perform a convolution operation on the channel that has been input with preprocessed data and assigned weights separately, then step S1033 is to fuse the feature maps of all channels to obtain the feature data of the preprocessed data, specifically by performing a weighted sum of the channel information after each depth convolution through the second convolution kernel, and then generating a new feature map. If step S1032 is to perform a convolution operation on the channel group that has been input with preprocessed data and assigned weights, then step S1033 is to splice the feature maps of all channels to obtain the feature data of the preprocessed data, specifically by combining the feature maps of different groups (usually splicing by channel dimension) to obtain the final output feature map, that is, each group of feature maps is independently processed by group convolution, and then the output feature maps of each group are merged. Therefore, the fusion of feature maps will not change the spatial dimension (height and width) of the feature map, but only change the number of channels, while the splicing of feature maps will increase the number of channels of the output feature map, but the spatial dimension remains unchanged.

[0103] As another embodiment of the present application, for different data types of preprocessed data, extracting feature data of the preprocessed data using the improved convolutional neural network can be implemented through steps S'1031 and S'1032, as described in detail as follows:

[0104] Step S'1031: Map the preprocessed data to the latent space after encoding to obtain latent features.

[0105] Specifically, as an embodiment of the present application, the preprocessed data is mapped to the latent space after encoding, and the latent features can be obtained by: mapping the input data to the latent space through an encoder based on a multi-layer neural network, and outputting the mean and logarithmic variance; introducing randomness into the mean and logarithmic variance to generate latent features.

[0106] Step S'1032: Decode the potential features and reconstruct the data through a preset loss function to obtain feature data of the preprocessed data.

[0107] Specifically, as an embodiment of the present application, the latent features are decoded and the data is reconstructed through a preset loss function, and the feature data of the preprocessed data can be: the reconstruction loss and KL divergence of the preset loss function, wherein the reconstruction loss is a measure of the difference between the original data and the reconstructed data (such as the mean square error), and the KL divergence is a measure of the difference between the latent distribution generated by the first encoder and the prior distribution; with the reconstruction loss and KL divergence as the measure of the total loss, the latent features are decoded and the data is reconstructed to obtain the feature data of the preprocessed data. Since the latent features generated by the encoder can be regarded as a low-dimensional feature representation of the data, which captures the key features of the data in the latent space, feature extraction can be performed.

[0108] As another embodiment of the present application, the latent features are decoded and the data is reconstructed by a preset loss function, and the feature data of the preprocessed data can be: the reconstruction loss and the sparsity regularization term of the preset loss function, wherein the reconstruction loss is a measure of the difference between the original data and the reconstructed data (such as the mean square error), and the sparsity regularization term is used to limit the average activation value of the second encoder activation function to be close to zero; with the reconstruction loss and the sparsity regularization term as the measure of the total loss, the latent features are decoded and the data is reconstructed to obtain the feature data of the preprocessed data. Since the latent features generated by the second encoder can be regarded as a low-dimensional feature representation of the data, which captures the key features of the data in the latent space, feature extraction can be performed.

[0109] As another embodiment of the present application, for different data types of preprocessed data, the feature data of the preprocessed data can be extracted using a preset algorithm based on deep learning. The preprocessed data is converted into an embedding sequence suitable for input into a bidirectional LSTM through an embedding layer; the embedded sequence is input into the bidirectional LSTM to obtain the hidden state of each time step, capturing the bidirectional dependency in the time series; the importance score of each hidden state is calculated through the attention mechanism, and the attention weight of each time step is generated through the Softmax function; the attention weight is used to perform weighted summation on the hidden states to generate a context vector, and the most representative global features are extracted as the feature data of the preprocessed data.

[0110] Step S104: Based on a machine learning algorithm, data mining is performed on the feature data of the preprocessed data to obtain data mining results.

[0111] In an embodiment of the present application, based on a machine learning algorithm, data mining is performed on the feature data of the preprocessed data to obtain data mining results. A support vector machine (SVM) model, a random forest model, or an improved K-means clustering algorithm is used to perform data mining on the feature data of the preprocessed data to obtain data mining results.

[0112] As an embodiment of the present application, a support vector machine SVM model is used to perform data mining on the feature data of the preprocessed data to obtain data mining results, which can be specifically implemented by steps Sa1041 to Sa1043, as described in detail as follows:

[0113] Step Sa1041: Calculate the kernel matrix for each kernel function to obtain a combined kernel matrix.

[0114] Assume that the feature data is X∈R N*d , where N is the number of samples, d is the feature dimension, and the label is y∈{+1,-1} N. You can choose multiple kernel functions k 1 , k 2 , ..., k m , such as linear kernel, polynomial kernel, radial basis function, etc. For each kernel function, calculate the kernel matrix K i (X,X) is as follows:

[0115] K i (X,X)=Kernel i (X,X)

[0116] According to the weight α of the i-th kernel function i , and the combined kernel matrix K(X,X) is obtained by weighted summation using the following formula:

[0117]

[0118] The weight α of the above i-th kernel function i It can be learned through optimization algorithms.

[0119] Step Sa1042: Use the combined kernel matrix to train the SVM model to obtain a trained SVM model.

[0120] It is understandable that the goal of the SVM model is to find an optimal hyperplane to maximize the margin of the classification boundary:

[0121]

[0122] subject to y i (ω·φ(x i )+b)≥1-ξ i ,ξ i ≥0

[0123] Where ω is the weight vector, φ(x i ) is the kernel function mapping, ξ i is the slack variable and C is the regularization parameter.

[0124] Step Sa1043: Input the feature data of the preprocessed data into the trained SVM model, and the output of the trained SVM model is used as the data mining result.

[0125] When the trained SVM model processes the feature data of the preprocessed data, the classification performance can be improved by adjusting the margins for the unbalanced data set. Specifically, it can be for the minority samples.

[0126] This assigns a higher weight ω i ,Right now Alternatively, use oversampling or undersampling technology to adjust the sample ratio in the data set to balance the category distribution and obtain reasonable data mining results. As an embodiment of the present application, a random forest model is used to perform data mining on the feature data of the preprocessed data to obtain data mining results. Specifically, the data mining results can be: randomizing the samples and features of the preprocessed data to obtain multiple sample subsets; using multiple sample subsets and training the random forest model based on the gradient boosting algorithm to obtain a trained random forest model; inputting the feature data of the preprocessed data into the trained random forest model, and the output of the trained random forest model is used as the data mining result.

[0127] As an embodiment of the present application, an improved K-means clustering algorithm is used to perform data mining on the feature data of the preprocessed data to obtain data mining results. The data mining results can be specifically implemented through steps Sb1041 to Sb1043, which are described in detail as follows:

[0128] Step Sb1041: Divide the feature data of the preprocessed data into multiple clusters and generate cluster centroids.

[0129] In an embodiment of the present application, if the preprocessed data belongs to a large data set, the preprocessed data can be pre-clustered to create an initial cluster that is more suitable for K-means; the data points are distributed into multiple clusters to generate smaller clusters as input data; the centroid is calculated for each cluster and used as the initial centroid candidate for K-means. The above scheme not only reduces the amount of data processed by the K-means clustering algorithm, but also reduces the computational burden of the K-means clustering algorithm. It should be noted that the feature data of the preprocessed data is divided into multiple clusters, and the number of clusters is usually much less than the K value of the K-means clustering algorithm.

[0130] Step Sb1042: Use a preset algorithm to calculate different K values ​​to obtain the optimal K value.

[0131] In the K-means clustering algorithm, selecting the best K value (the so-called K value refers to the number of clusters, that is, K value = number of clusters) is a key step in determining the number of clusters, which ensures the best effect of the K-means clustering algorithm. As an embodiment of the present application, a scheme for calculating different K values ​​using a preset algorithm to obtain the best K value can be: for different K values, run the K-means clustering algorithm, calculate the total cluster sum of squares error (Within-Cluster Sum of Squares, WCSS) for each K value; draw a relationship graph between the K value and the WCSS, and select the elbow point as the best K value, where the calculation formula of WCSS is as follows:

[0132]

[0133] Among them, C i is the sample of the ith cluster, μ i is the centroid of the ith cluster.

[0134] As another embodiment of the present application, a scheme for calculating different K values ​​using a preset algorithm to obtain the optimal K value may be: for different K values, calculating the silhouette coefficient; selecting the K value with the highest average silhouette coefficient as the optimal K value, wherein the calculation formula of the silhouette coefficient is as follows:

[0135]

[0136] Among them, a(i) is the average distance from sample i to other samples in the same cluster, and b(i) is the average distance from sample i to the nearest sample in a different cluster.

[0137] Step Sb1043: Cluster multiple clusters according to the optimal K value and cluster centroids to obtain data mining results.

[0138] Specifically, the optimal K value is used as the number of clusters of the K-means clustering algorithm; the cluster centroid generated by step Sb1041 is used as the initial centroid of the K-means clustering algorithm to generate fewer preliminary clusters; the data point is assigned to the cluster where the centroid closest to it is located, and the centroid starts from the cluster centroid generated by step Sb1041; the new centroid is iteratively calculated, and the centroid of each cluster is updated; the assignment and update steps are continued until the centroid is stable or the maximum number of iterations is reached, and the clustering result is obtained as the data mining result, wherein the iterative calculation of the new centroid and the updating of the centroid of each cluster mainly include the following steps: The distribution stage is mainly to calculate the distance from each data point to all K centroids, select the cluster corresponding to the nearest centroid, update the cluster label of each data point, and form a new cluster distribution; the update stage is mainly to calculate the mean of all data points in each cluster, take it as the new centroid, and update the centroid position of all clusters; and the convergence check stage is mainly to determine whether the change of the centroid is less than the set threshold or reaches the maximum number of iterations. If the centroid changes very little, it means that the algorithm has converged and the process of K-means clustering algorithm ends.

[0139] Step S105: Display the data mining results to the user in a visual manner.

[0140] In order to facilitate users to understand the mined results, the data mining results are displayed to users in a visual way, including dynamic interactive dashboards, multidimensional data display and text visualization. Among them, the dynamic interactive dashboard uses front-end visualization libraries such as D3.js and Plotly to provide real-time updates and interactive functions. Users can click, zoom and hover to view detailed information; multidimensional data display refers to the display of high-dimensional data and multivariate relationships through parallel coordinates and scatter matrix diagrams; and text visualization is for text data, mainly displaying Word Cloud, Topic Modeling and sentiment analysis results, combining color and size to highlight the key topics and sentiment tendencies in the text.

[0141] From the above attached Figure 1 From the network data mining method of the example, it can be seen that after collecting data from different data sources, the data collected from different data sources are preprocessed to obtain the preprocessed data; for different data types of the preprocessed data, the preset algorithm based on deep learning is used to extract the feature data of the preprocessed data; based on the machine learning algorithm, the feature data of the preprocessed data is mined to obtain the data mining results. Since the preset algorithm based on deep learning is suitable for data of various types or patterns and can accurately extract data features, and the machine learning algorithm shows excellent performance in data classification, and these algorithms are all executed by computer programs, even for large-scale data, the task can be executed in real time, that is, it has strong real-time performance.

[0142] Please see attached Figure 2 , is a network data mining device provided in an embodiment of the present application, the device may include a collection module 201, a preprocessing module 202, an extraction module 203, a mining module 204 and a display module 205, which are described in detail as follows:

[0143] The acquisition module 201 is used to acquire data from different data sources;

[0144] A preprocessing module 202 is used to preprocess the data collected from different data sources to obtain preprocessed data;

[0145] An extraction module 203 is used to extract feature data of the preprocessed data using a preset algorithm based on deep learning for different data types of the preprocessed data;

[0146] A mining module 204 is used to perform data mining on the feature data of the preprocessed data based on a machine learning algorithm to obtain a data mining result;

[0147] The display module 205 is used to display the data mining results to the user in a visual manner.

[0148] From the above attached Figure 2 It can be seen from the example network data mining device that after collecting data from different data sources, the data collected from different data sources are preprocessed to obtain preprocessed data; for different data types of the preprocessed data, a preset algorithm based on deep learning is used to extract feature data of the preprocessed data; based on the machine learning algorithm, data mining is performed on the feature data of the preprocessed data to obtain data mining results. Since the preset algorithm based on deep learning is suitable for data of various types or patterns and can accurately extract data features, and the machine learning algorithm shows excellent performance in data classification, and these algorithms are all executed by computer programs, even for large-scale data, the task can be executed in real time, that is, it has strong real-time performance.

[0149] Figure 3 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 3 As shown, the electronic device 3 of this embodiment mainly includes: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30, such as a program of a network data mining method. When the processor 30 executes the computer program 32, the steps in the above network data mining method embodiment are implemented, such as Figure 1 Alternatively, when the processor 30 executes the computer program 32, the functions of each module / unit in the above-mentioned device embodiments are implemented, for example Figure 2 The functions of the collection module 201, the preprocessing module 202, the extraction module 203, the mining module 204 and the display module 205 are shown.

[0150] Exemplarily, the computer program 32 of the network data mining method mainly includes: collecting data from different data sources; preprocessing the data collected from different data sources to obtain preprocessed data; for different data types of the preprocessed data, using a preset algorithm based on deep learning to extract feature data of the preprocessed data; based on a machine learning algorithm, performing data mining on the feature data of the preprocessed data to obtain data mining results; and presenting the data mining results to the user in a visual manner. The computer program 32 can be divided into one or more modules / units, one or more modules / units are stored in the memory 31, and are executed by the processor 30 to complete the present application. One or more modules / units can be a series of computer program instruction segments that can perform specific functions, and the instruction segments are used to describe the execution process of the computer program 32 in the electronic device 3. For example, the computer program 32 can be divided into the functions of an acquisition module 201, a preprocessing module 202, an extraction module 203, a mining module 204 and a display module 205 (modules in the virtual device), and the specific functions of each module are as follows: the acquisition module 201 is used to collect data from different data sources; the preprocessing module 202 is used to preprocess the data collected from different data sources to obtain preprocessed data; the extraction module 203 is used to extract feature data of the preprocessed data using a preset algorithm based on deep learning for different data types of the preprocessed data; the mining module 204 is used to perform data mining on the feature data of the preprocessed data based on a machine learning algorithm to obtain data mining results; the display module 205 is used to display the data mining results to the user in a visual manner.

[0151] The electronic device 3 may include but is not limited to a processor 30 and a memory 31. Those skilled in the art will appreciate that Figure 3 It is only an example of electronic device 3 and does not constitute a limitation of electronic device 3. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, a computing device may also include input and output devices, network access devices, buses, etc.

[0152] The processor 30 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0153] The memory 31 may be an internal storage unit of the electronic device 3, such as a hard disk or memory of the electronic device 3. The memory 31 may also be an external storage device of the electronic device 3, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 3. Further, the memory 31 may also include both an internal storage unit of the electronic device 3 and an external storage device. The memory 31 is used to store computer programs and other programs and data required by the electronic device. The memory 31 may also be used to temporarily store data that has been output or is to be output.

[0154] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0155] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0156] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0157] In the embodiments provided in the present application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0158] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0159] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0160] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program of the network data mining method can be stored in a storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments, that is, collect data from different data sources; pre-process the data collected from different data sources to obtain pre-processed data; for different data types of the pre-processed data, use a preset algorithm based on deep learning to extract the feature data of the pre-processed data; based on a machine learning algorithm, perform data mining on the feature data of the pre-processed data to obtain data mining results; and display the data mining results to the user in a visual manner. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form. Storage media may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, mobile hard disks, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content contained in the storage medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, storage media do not include electric carrier signals and telecommunication signals.

[0161] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application is described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. The specific implementation methods described above further describe the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation method of the present application, and is not used to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present invention.

Claims

1. A network data mining method, characterized in that: The method comprises: Collect data from different data sources; Preprocessing the data collected from different data sources to obtain preprocessed data; For different data types of the preprocessed data, extract feature data of the preprocessed data using a preset algorithm based on deep learning; Based on a machine learning algorithm, data mining is performed on the feature data of the preprocessed data to obtain a data mining result; The data mining results are displayed to the user in a visual manner.

2. The network data mining method according to claim 1, characterized in that: The preprocessing of the data collected from different data sources to obtain preprocessed data of the data includes: Cleaning the data collected from different data sources to obtain cleaned data; Use any one of a Gaussian filtering algorithm, a moving average method and an entropy filtering algorithm or a combination thereof to remove noise from the cleaned data to obtain denoised data; The denoised data is normalized to obtain preprocessed data of the data.

3. The network data mining method according to claim 1, characterized in that: The method of extracting feature data of the preprocessed data using a preset algorithm based on deep learning for different data types of the preprocessed data includes: For different data types of the preprocessed data, using an improved convolutional neural network to extract feature data of the preprocessed data; or According to different data types of the preprocessed data, an automatic encoder is used to extract feature data of the preprocessed data.

4. The network data mining method according to claim 3, characterized in that: The method of extracting feature data of the preprocessed data using an improved convolutional neural network for different data types of the preprocessed data includes: Use a specific activation function and generate the weight of each channel based on the attention mechanism and assign it to the corresponding channel; Performing convolution operation on the channels that input the preprocessed data and assign weights, individually or in groups, to generate corresponding feature maps; The feature maps of all channels are fused or concatenated to obtain the feature data of the preprocessed data.

5. The network data mining method according to claim 3, characterized in that: The method of extracting feature data of the preprocessed data using an automatic encoder for different data types of the preprocessed data includes: Mapping the preprocessed data to a latent space after encoding to obtain latent features; The potential features are decoded and data is reconstructed through a preset loss function to obtain feature data of the preprocessed data.

6. The network data mining method according to claim 1, characterized in that: The data mining is performed on the feature data of the preprocessed data based on the machine learning algorithm to obtain the data mining results, including: A support vector machine (SVM) model, a random forest model or an improved K-means clustering algorithm is used to perform data mining on the feature data of the preprocessed data to obtain a data mining result.

7. The network data mining method according to claim 6, characterized in that: The method adopts a support vector machine (SVM) model to perform data mining on the feature data of the preprocessed data to obtain a data mining result, including: calculating a kernel matrix for each kernel function to obtain a combined kernel matrix; training an SVM model using the combined kernel matrix to obtain a trained SVM model; inputting the feature data of the preprocessed data into the trained SVM model, and outputting the trained SVM model as the data mining result; The method of using a random forest model to perform data mining on the feature data of the preprocessed data to obtain a data mining result includes: performing sample and feature randomization on the preprocessed data to obtain a plurality of sample subsets; using the plurality of sample subsets and training the random forest model based on a gradient boosting algorithm to obtain a trained random forest model; inputting the feature data of the preprocessed data into the trained random forest model, and using the output of the trained random forest model as the data mining result; The improved K-means clustering algorithm is used to perform data mining on the feature data of the preprocessed data to obtain data mining results, including: dividing the feature data of the preprocessed data into multiple clusters to generate cluster centroids; using a preset algorithm to calculate different K values ​​to obtain an optimal K value; clustering the multiple clusters according to the optimal K value and the cluster centroids to obtain the data mining results.

8. A network data mining device, characterized in that: The device comprises: A collection module, used to collect data from different data sources; A preprocessing module, used for preprocessing the data collected from different data sources to obtain preprocessed data of the data; An extraction module, for extracting feature data of the preprocessed data using a preset algorithm based on deep learning for different data types of the preprocessed data; A mining module, used for performing data mining on the feature data of the preprocessed data based on a machine learning algorithm to obtain a data mining result; The display module is used to display the data mining results to the user in a visual manner.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images