Multi-dimensional data analysis method and system based on deep learning
Through deep learning technology, preprocessing, feature extraction and dimensionality reduction of multidimensional data, the problems of high computing cost, slow processing speed and low accuracy in existing multidimensional data analysis are solved, and fast and accurate data recognition and real-time processing are achieved. Enhanced interpretability of the results.
Patent Information
- Application Number
- CN202410930251.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-11
- Publication Date
- 2024-11-12
AI Technical Summary
The existing multidimensional data analysis technology has high calculation cost, slow processing speed and low accuracy when processing large-scale and high-dimensional data. It cannot meet the real-time processing needs and is difficult to interpret the results, affecting the accuracy and reliability of the identification.
A multidimensional data analysis method based on deep learning is adopted to train models through data preprocessing, feature extraction and data dimensionality reduction, and use gradient descent algorithms to obtain data recognition models, which are used to quickly and accurately identify multidimensional data.
It improves the accuracy and efficiency of data identification, reduces calculation costs, realizes real-time processing, enhances the interpretability of results, and avoids the limitations and misleading of traditional methods.
Smart Images

Figure CN118940101A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a multidimensional data analysis method based on deep learning and a multidimensional data analysis system based on deep learning. Background Art
[0002] In order to solve the problem of identifying large amounts of data in multidimensional data analysis, current existing technologies mainly use traditional statistical methods or machine learning methods, such as data identification technology based on clustering algorithms.
[0003] This prior art divides the data set into multiple different groups according to certain rules from the perspective of clustering, thereby realizing the classification and identification of data. This technology is usually implemented using algorithms such as k-means clustering and hierarchical clustering, among which k-means clustering is one of the more common methods. Specifically, k-means clustering divides the data set into k different clusters, each cluster represents a collection of data, and determines the ownership of the data by calculating the distance from the data point to the cluster center point. The advantage of this algorithm is that it has high efficiency and accuracy, and is easy to implement.
[0004] However, this prior art has some defects. First, since algorithms such as k-means clustering are sensitive to the initial cluster center selection, the results of the algorithm will be affected by the initial value. Secondly, this technology has high requirements for data distribution and has poor recognition effect on non-convex or noisy data. At present, there is an increasing demand to extract features from a large amount of data and use them to identify and predict future data. Although the prior art uses multidimensional data analysis technology for identification, it has the following problems when processing large-scale, high-dimensional data:
[0005] 1) High computational cost: The multidimensional data analysis algorithm used in existing technologies requires a large amount of calculation and places high demands on computer performance, resulting in high computational costs.
[0006] 2) Slow processing speed: Since the existing technology needs to process large-scale, high-dimensional data, it takes a lot of time to complete data processing and analysis, resulting in slow processing speed.
[0007] 3) Low accuracy: The algorithms used in the prior art do not have high accuracy in data classification and recognition, and there are certain errors and misjudgment rates, which affects the accuracy and reliability of recognition.
[0008] 4) Inability to perform real-time processing: Due to the slow processing speed and high computing cost of existing technologies, they cannot meet the needs of real-time processing, limiting their application in certain application scenarios.
[0009] 5) Difficulty in explaining results: The multidimensional data analysis algorithms used in the prior art are often difficult to explain their results, resulting in low user confidence in the results and failure to meet the interpretability requirements of the algorithms.
[0010] To address these problems, a new multidimensional data analysis solution based on deep learning needs to be proposed. Summary of the invention
[0011] The purpose of the embodiments of the present invention is to provide a multidimensional data analysis method and system based on deep learning, so as to at least solve the problems of low recognition efficiency and low accuracy of existing multidimensional data recognition schemes.
[0012] In order to achieve the above-mentioned objectives, the first aspect of the present invention provides a multidimensional data analysis method based on deep learning, the method comprising: collecting a multidimensional data set where the target data is located, and performing preprocessing on the multidimensional data set; performing feature extraction and data dimensionality reduction processing on the preprocessed multidimensional data set in sequence; based on the reduced dimensionality data and extracted features, performing model training in a preselected classification model to obtain a data recognition model; using the data to be recognized as input parameters of the data recognition model, performing model training, and judging whether the data to be recognized is the target data based on the training results.
[0013] Optionally, the preprocessing of the multidimensional data set includes: sequentially performing data deduplication, missing value filling, outlier processing and data format conversion on the multidimensional data set to obtain a preprocessed multidimensional data set.
[0014] Optionally, the processing rules for performing feature extraction on the preprocessed multidimensional data set are: classify each data in the preprocessed multidimensional data set to obtain a classified data set corresponding to each data classification; wherein the data classification includes: high-dimensional data, speech signal data, image data and time series data whose dimensions are greater than a preset dimensionality threshold; based on the preset feature extraction algorithm corresponding to each data classification, perform parallel feature extraction on each classified data set.
[0015] Optionally, the preset feature extraction algorithm for high-dimensional data whose dimension is greater than a preset dimensionality threshold is a principal component analysis algorithm; the feature extraction algorithm for the speech signal data is an independent component analysis algorithm; and the feature extraction algorithm for the image data and time series data is a wavelet transform algorithm.
[0016] Optionally, after performing feature extraction on the preprocessed multidimensional data set, the method further includes: determining the most representative feature, including: identifying the most representative feature among the extracted features based on a filtering method or a wrapping method, and marking the identified most representative feature as a key feature.
[0017] Optionally, data dimension reduction is performed on the preprocessed multidimensional data set, including: performing data dimension reduction on the feature extraction or feature set based on a principal component analysis algorithm or a linear discriminant analysis algorithm to obtain a feature subset after dimension reduction; wherein, the processing rules for performing data dimension reduction on the feature extraction or feature set based on the principal component analysis algorithm to obtain the feature subset after dimension reduction are: performing standardization on each feature in the feature set, and calculating the covariance matrix of each feature after standardization; performing eigenvalue decomposition on the covariance matrix to obtain eigenvectors and corresponding eigenvalues; sorting the eigenvectors according to the size of the corresponding eigenvalues, selecting the top K eigenvectors, and constructing a feature subset after dimension reduction.
[0018] Optionally, data dimension reduction is performed on the feature extraction or feature set based on a linear discriminant analysis algorithm to obtain a feature subset after dimension reduction, including: calculating the mean vector of each type of feature respectively, and calculating the overall mean vector; calculating the intra-class scatter matrix and the inter-class scatter matrix based on the mean vector of each type of feature and the overall mean vector, and combining the intra-class scatter matrix and the inter-class scatter matrix to obtain a combined matrix; performing eigenvalue decomposition on the combined matrix to obtain eigenvectors and eigenvalues; sorting the eigenvectors according to the size of the corresponding eigenvalues, selecting the first K eigenvectors, and constructing a feature subset after dimension reduction.
[0019] Optionally, based on the reduced-dimensional data and extracted features, model training is performed in a preselected classification model to obtain a data recognition model, including: performing model training on the reduced-dimensional data and extracted features based on any one of the model training algorithms, namely, the gradient descent algorithm, the stochastic gradient descent algorithm and the Adam algorithm, to obtain an initial model; based on the reserved reduced-dimensional data and the extracted features, optimizing various preset optimization objectives of the initial model to obtain an optimized model as a data recognition model; wherein the preset optimization objectives include: regularization, hyperparameter tuning, model fusion, data enhancement and network structure optimization.
[0020] The second aspect of the present invention provides a multidimensional data analysis system based on deep learning, the system comprising: an acquisition unit, used to acquire the multidimensional data set where the target data is located, and to perform preprocessing on the multidimensional data set; a dimensionality reduction unit, used to sequentially perform feature extraction and data dimensionality reduction processing on the preprocessed multidimensional data set; a training unit, used to perform model training in a preselected classification model based on the dimensionality reduced data and extracted features to obtain a data recognition model; an identification unit, used to use the data to be identified as an input parameter of the data recognition model, perform model training, and determine whether the data to be identified is the target data based on the training results.
[0021] On the other hand, the present invention provides a computer-readable storage medium, which stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned multidimensional data analysis method based on deep learning.
[0022] Through the above technical solutions, using multiple data dimensions for analysis and identification can provide more accurate results than traditional single data sources, avoiding the limitations and misleading nature of single-dimensional data. Because this technology can utilize multiple data sources and adopt effective algorithms and parallel computing technologies, it can process large-scale data sets and provide fast and accurate results. It automatically adjusts algorithms and parameters when facing different types of data, thereby improving the ability and efficiency of processing different data types.
[0023] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:
[0025] Figure 1 is a flowchart of the steps of a multidimensional data analysis method based on deep learning provided by an embodiment of the present invention;
[0026] Figure 2 It is a system structure diagram of a multidimensional data analysis system based on deep learning provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0027] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.
[0028] In modern society, data analysis is very important. Data analysis can be used in different fields such as business, healthcare, finance, science, etc. Through data analysis, we can obtain a lot of information and gain valuable insights from it. An important field of data analysis is data classification and prediction. In data classification and prediction, we need to extract features from a large amount of data and use them to identify and predict future data.
[0029] In the past, data classification and prediction were often done through manual analysis and decision-making. However, this approach has many limitations. First, it takes a lot of time and effort to complete. Second, it usually requires expertise and experience. Finally, human decision makers may be affected by emotions and subjective biases, resulting in inaccurate or unreliable decisions.
[0030] To overcome these limitations, automated data classification and prediction techniques have been widely used. In these techniques, computer programs can automatically extract features from data and use these features to identify and predict future data. These techniques are usually based on machine learning and artificial intelligence techniques, such as neural networks, decision trees, support vector machines, etc.
[0031] Although these techniques have achieved some success, they still have some limitations. First, they can usually only handle one or a few types of data. Second, they can usually only handle a small number of features. Finally, they can usually only handle low-dimensional data, that is, only a few features. Therefore, a new technique is needed to overcome these limitations and achieve more accurate and reliable data classification and prediction.
[0032] In order to solve the problem of identifying a large amount of data in multidimensional data analysis, the current prior art mainly adopts traditional statistical methods or machine learning methods, among which the prior art closest to the present invention is the data identification technology based on clustering algorithm.
[0033] This prior art divides the data set into multiple different groups according to certain rules from the perspective of clustering, thereby realizing the classification and identification of data. This technology is usually implemented using algorithms such as k-means clustering and hierarchical clustering, among which k-means clustering is one of the more common methods. Specifically, k-means clustering divides the data set into k different clusters, each cluster represents a collection of data, and determines the ownership of the data by calculating the distance from the data point to the cluster center point. The advantage of this algorithm is that it has high efficiency and accuracy, and is easy to implement.
[0034] However, this prior art has some defects. First, since algorithms such as k-means clustering are sensitive to the initial cluster center selection, the results of the algorithm will be affected by the initial value. Second, this technology has high requirements for data distribution and has poor recognition effect on non-convex or noisy data.
[0035] Therefore, the present invention proposes a multidimensional data analysis and recognition technology based on deep learning, which uses a deep learning model to analyze and recognize a large amount of data, avoiding the tedious process of manually selecting features in traditional methods, and can process nonlinear and high-dimensional data. Compared with the prior art, the technology of the present invention can better solve the data recognition problem and has higher accuracy and stability.
[0036] Although the existing technology uses multidimensional data analysis technology for identification, it has the following problems when processing large-scale, high-dimensional data:
[0037] High computational cost: The multidimensional data analysis algorithms used in existing technologies require a large amount of calculations, which places high demands on computer performance, resulting in high computational costs.
[0038] Slow processing speed: Since existing technologies need to process large-scale, high-dimensional data, it takes a lot of time to complete data processing and analysis, resulting in slow processing speed.
[0039] Low accuracy: The algorithms used in the existing technology do not have high accuracy in data classification and recognition. There are certain errors and misjudgment rates, which affect the accuracy and reliability of recognition.
[0040] Inability to perform real-time processing: Due to the slow processing speed and high computing cost of existing technologies, they cannot meet the needs of real-time processing, limiting their application in certain application scenarios.
[0041] Difficulty in explaining results: The multidimensional data analysis algorithms used in existing technologies are often difficult to explain their results, resulting in low user confidence in the results and failure to meet the interpretability requirements of the algorithms.
[0042] Figure 1 1 is a flowchart of a multidimensional data analysis method based on deep learning provided by an embodiment of the present invention. Figure 1 As shown, an embodiment of the present invention provides a multidimensional data analysis method based on deep learning, the method comprising:
[0043] Step S10: Collect the multidimensional data set where the target data is located, and perform preprocessing on the multidimensional data set.
[0044] Specifically, a multidimensional dataset usually contains data from multiple sources, which may include relational databases, data warehouses, cloud storage, IoT devices, etc. The purpose of data collection is to bring these scattered data into a unified multidimensional dataset for subsequent processing and analysis. This dataset can be structured (such as a table in a database) or unstructured (such as text, images, videos, etc.). The collected data should cover all dimensions and features of the target data for subsequent analysis and identification.
[0045] Furthermore, data deduplication is the primary step of preprocessing. Duplicate data will not only lead to deviations in the analysis results, but also waste storage resources and computing resources. The purpose of data deduplication is to identify and delete duplicate records in a multidimensional data set, thereby ensuring that each record exists uniquely in the data set. There are many ways to implement data deduplication, and appropriate algorithms and tools can be selected according to specific circumstances. The scheme of the present invention includes primary key-based deduplication, similarity-based deduplication, and the like. Primary key-based deduplication is the simplest method, which identifies records with the same primary key in the data set and retains one record, while deleting the rest. Similarity-based deduplication is more complicated, and usually requires the use of a machine learning algorithm to determine the similarity between records to identify duplicate records.
[0046] Furthermore, the scheme of the present invention includes deleting records with missing values, filling missing values with mean or median, predicting missing values based on regression model, etc. Deleting records with missing values is suitable for situations where there are fewer missing values. If there are too many missing values, deleting records may result in insufficient data, thereby affecting the representativeness of the analysis results. Filling missing values with mean or median is suitable for numerical data. This method is simple and easy to implement, but it may obscure the true distribution of the data. Predicting missing values based on regression model is a more sophisticated method, which can predict missing values based on the relationship between data, thereby improving the accuracy of filling.
[0047] Furthermore, there are two main methods for outlier processing in the scheme of the present invention: deleting outliers and adjusting outliers. Deleting outliers is suitable for situations where there are few outliers and they are obviously wrong. By identifying and deleting outliers, their interference with the analysis results can be eliminated. Adjusting outliers is suitable for situations where there are many outliers or they are difficult to delete. By appropriately adjusting the outliers, the outliers can be close to the normal range, thereby reducing their impact on the analysis results. Adjustment methods include tailing method, data transformation method, etc.
[0048] Furthermore, methods of data format conversion include type conversion, format standardization, etc. Type conversion refers to converting data from one type to another, such as converting text data to numerical data for numerical calculation and statistical analysis.
[0049] Based on the solution of the present invention, due to the complexity and diversity of multidimensional data, there may be a lot of incomplete, repeated, useless or erroneous information in the data. Therefore, before data analysis and recognition, the data needs to be cleaned and preprocessed to ensure the quality and accuracy of the data. This includes data deduplication, missing value filling, outlier processing, data format conversion, etc.
[0050] Step S20: performing feature extraction and data dimensionality reduction processing on the preprocessed multidimensional data set in sequence.
[0051] Specifically, the processing rules for performing feature extraction on the preprocessed multidimensional data set are: classify each data in the preprocessed multidimensional data set to obtain a classified data set corresponding to each data classification; wherein the data classification includes: high-dimensional data, speech signal data, image data and time series data whose dimensions are greater than a preset dimensionality threshold; based on the preset feature extraction algorithm corresponding to each data classification, perform parallel feature extraction on each classified data set.
[0052] In the embodiment of the present invention, data classification is the first step in the feature extraction process, and its purpose is to separate different types of data so that the most appropriate feature extraction algorithm can be used for each data type. Specifically, the process of data classification is as follows:
[0053] 1) High-dimensional data: High-dimensional data usually refers to a data set whose number of dimensions is greater than a preset threshold. This type of data has a high dimensionality in the feature space, and direct processing will lead to high computational complexity and the "curse of dimensionality" problem. Typical high-dimensional data include gene expression data, text data, etc.
[0054] 2) Speech signal data: This type of data is usually a sound signal captured by a recording device and has the characteristics of a time series. Speech signal data needs to go through specific preprocessing steps, such as noise removal and signal enhancement, to extract useful features.
[0055] 3) Image data: Image data is a two-dimensional or three-dimensional pixel matrix that contains rich visual information. Feature extraction of image data usually needs to consider spatial information and texture information.
[0056] 4) Time series data: Time series data is observation data arranged in chronological order. The feature extraction of this type of data needs to consider time correlation and trend analysis.
[0057] Furthermore, the preset feature extraction algorithm for high-dimensional data whose dimension is greater than a preset dimensionality threshold is a principal component analysis algorithm; the feature extraction algorithm for the speech signal data is an independent component analysis algorithm; and the feature extraction algorithm for the image data and time series data is a wavelet transform algorithm.
[0058] In an embodiment of the present invention, for high-dimensional data whose dimension is greater than a preset dimension threshold, the feature extraction algorithm selected by the scheme of the present invention is principal component analysis (PCA). PCA is a statistical method for converting a high-dimensional data set into a low-dimensional data set while retaining as much original data variation information as possible. The specific process is to calculate the mean of the data set and subtract it from the original data to centralize the data. The covariance matrix is calculated by centralizing the data to capture the linear relationship in the data. The covariance matrix is subjected to eigenvalue decomposition to obtain eigenvalues and eigenvectors. The eigenvectors corresponding to the first k largest eigenvalues are selected as principal components to form a new low-dimensional feature space. The main advantage of PCA is that it can reduce the dimension of the data, reduce the computational complexity, and the extracted principal components can explain most of the variation in the data. Feature extraction by PCA can significantly improve the efficiency and effect of subsequent model training.
[0059] Furthermore, for speech signal data, the scheme of the present invention selects Independent Component Analysis (ICA) for feature extraction. ICA is a statistical technique for decomposing multivariate signals into independent non-Gaussian signals. The specific process is to center and whiten the speech signal to eliminate correlation. The independent components are found by maximizing the non-Gaussianity through an iterative algorithm. The original signal is reconstructed using independent components to extract useful features. The main applications of ICA in speech signal processing include blind source separation and feature extraction. Through ICA, the independent sources in the speech signal can be effectively separated, and features useful for speech recognition and sentiment analysis can be extracted.
[0060] Furthermore, for image data, wavelet transform (WT) is an effective feature extraction algorithm. Wavelet transform can decompose an image into sub-bands of different scales and positions to capture local features and multi-scale characteristics in the image. The specific process is to select appropriate wavelet basis functions according to the image characteristics, such as Haar wavelet, Daubechies wavelet, etc. The image is decomposed into multiple sub-bands, including low-frequency sub-bands and high-frequency sub-bands through wavelet transform. Features such as energy, mean and variance are extracted from different sub-bands. The advantage of wavelet transform is that it can process the spatial and frequency information of the image at the same time, which is particularly suitable for applications such as edge detection, texture analysis and image compression.
[0061] Furthermore, for time series data, wavelet transform is also an effective feature extraction algorithm. Wavelet transform can decompose time series into different frequency components to analyze local characteristics and trends in time series. The specific process is to select appropriate wavelet basis functions according to the characteristics of time series. The time series is decomposed into multiple sub-bands by wavelet transform to capture features on different time scales. Features are extracted from different sub-bands, such as the energy, mean and rate of change of wavelet coefficients. The advantage of wavelet transform in time series analysis is that it can process non-stationary signals and is suitable for applications such as financial data, meteorological data and physiological signals. Features extracted by wavelet transform can help identify periodicity, mutation points and trend changes in time series.
[0062] Preferably, after performing feature extraction on the preprocessed multidimensional data set, the method further includes: determining the most representative feature, including: identifying the most representative feature among the extracted features based on a filtering method or a wrapping method, and marking the identified most representative feature as a key feature.
[0063] In the embodiment of the present invention, the most representative feature refers to the feature that is evaluated as the most contributing to the prediction or classification task during the feature selection process. It is usually able to provide the greatest information gain or most effectively reduce the prediction error of the model. The retained features are usually the most representative and can help understand the decision-making process of the model and increase the transparency and interpretability of the model.
[0064] Furthermore, data dimension reduction is performed on the preprocessed multidimensional data set, including: performing data dimension reduction on the feature extraction or feature set based on a principal component analysis algorithm or a linear discriminant analysis algorithm to obtain a feature subset after dimension reduction; wherein, the processing rules for performing data dimension reduction on the feature extraction or feature set based on the principal component analysis algorithm to obtain the feature subset after dimension reduction are: performing standardization on each feature in the feature set, and calculating the covariance matrix of each feature after standardization; performing eigenvalue decomposition on the covariance matrix to obtain eigenvectors and corresponding eigenvalues; sorting the eigenvectors according to the size of the corresponding eigenvalues, selecting the first K eigenvectors, and constructing a feature subset after dimension reduction.
[0065] In a possible implementation, PCA (Principal Component Analysis) is a classic dimensionality reduction technique that transforms data from the original high-dimensional space to a new low-dimensional space through a linear transformation of the original data, so that each feature in the new feature space is independent of each other, thereby reducing redundant information between features. The specific implementation process is as follows:
[0066] 1) Standardize the features, that is, set the mean of each feature to 0 and the variance to 1.
[0067] 2) Calculate the covariance matrix, that is, the correlation between features.
[0068] 3) Perform eigenvalue decomposition on the covariance matrix to obtain eigenvectors and eigenvalues.
[0069] 4) Sort the feature vectors according to the corresponding eigenvalues, select the first k feature vectors, and form a new feature subset.
[0070] Furthermore, based on the linear discriminant analysis algorithm, data dimension reduction processing is performed on the feature extraction or feature set to obtain a feature subset after dimension reduction, including: respectively calculating the mean vector of each type of feature, and calculating the overall mean vector; calculating the intra-class scatter matrix and the inter-class scatter matrix based on the mean vector of each type of feature and the overall mean vector, and combining the intra-class scatter matrix and the inter-class scatter matrix to obtain a combined matrix; performing eigenvalue decomposition on the combined matrix to obtain eigenvectors and eigenvalues; sorting the eigenvectors according to the size of the corresponding eigenvalues, selecting the first K eigenvectors, and constructing a feature subset after dimension reduction.
[0071] In a possible implementation, LDA (Linear Discriminant Analysis) is also a commonly used dimensionality reduction technique. The difference between it and PCA is that PCA is an unsupervised learning algorithm that only considers the disorder between samples; while LDA is a supervised learning algorithm that considers not only the distance between samples but also the category information, thereby better preserving the differences between samples during the dimensionality reduction process. The specific implementation process is as follows:
[0072] 1) For each category, calculate the mean vector of the category.
[0073] 2) Calculate the overall mean vector.
[0074] 3) Calculate the intra-class scatter matrix (Sw) and the inter-class scatter matrix (Sb).
[0075] 4) Perform eigenvalue decomposition on the matrix (Sw^-1)Sb to obtain eigenvectors and eigenvalues.
[0076] 5) Sort the feature vectors according to the corresponding eigenvalues, select the first k feature vectors, and form a new feature subset.
[0077] Through the above steps, the present invention can effectively extract and reduce the features of the original data, obtain a more concise and effective feature subset, and provide a better basis for subsequent classification and recognition.
[0078] Step S30: Based on the dimensionally reduced data and extracted features, model training is performed in a pre-selected classification model to obtain a data recognition model.
[0079] Specifically, model training is performed on the reduced-dimensional data and extracted features based on any one of the model training algorithms, namely, the gradient descent algorithm, the stochastic gradient descent algorithm and the Adam algorithm, to obtain an initial model; based on the reserved reduced-dimensional data and the extracted features, various preset optimization objectives of the initial model are optimized to obtain an optimized model as a data recognition model; wherein the preset optimization objectives include: regularization, hyperparameter tuning, model fusion, data enhancement and network structure optimization.
[0080] In the embodiment of the present invention, after data preprocessing and feature extraction, a classification model needs to be established to identify the target data. The model can be a classification model based on machine learning, such as a support vector machine, a decision tree, a random forest, etc., or a classification model based on deep learning, such as a convolutional neural network, a recurrent neural network, an autoencoder, etc. During the model training process, the model needs to be optimized and adjusted to obtain better performance and accuracy. The optimization objectives include:
[0081] 1) Regularization: Regularizing the model can prevent overfitting and reduce generalization error. L1, L2 regularization and Dropout are commonly used regularization techniques.
[0082] 2) Hyperparameter tuning: There are many hyperparameters that need to be adjusted in model training, such as learning rate, batch size, activation function, etc. The optimal hyperparameters can be found using techniques such as cross-validation.
[0083] 3) Model fusion: By fusing multiple different models together, the performance and accuracy of the model can be improved. Commonly used model fusion techniques include ensemble learning, stacked generalization, etc.
[0084] 4) Data enhancement: By enhancing the training data, the data set can be expanded and overfitting can be reduced. Data enhancement methods include rotation, flipping, translation, scaling, cropping, etc.
[0085] 5) Network structure optimization: The performance and accuracy of the model can be further improved by adjusting the network structure. Common network structure optimization techniques include convolution kernel optimization, network pruning, network distillation, etc.
[0086] Step S40: using the data to be identified as an input parameter of the data identification model, executing model training, and determining whether the data to be identified is target data based on the training result.
[0087] Specifically, based on the above steps, a classification model that can quickly identify target data is obtained. When unknown data is subsequently identified, it can be determined whether the unknown data is target data based on the model. The data to be identified is used as the input parameter of the data identification model, and model training is performed. Based on the training results, it is determined whether the data to be identified is target data. Based on this, the identification and classification of the data to be identified can be completed efficiently and accurately.
[0088] Figure 2 1 is a system structure diagram of a multidimensional data analysis system based on deep learning provided by an embodiment of the present invention. Figure 2 As shown, an embodiment of the present invention provides a multidimensional data analysis system based on deep learning, and the system includes: an acquisition unit, used to acquire a multidimensional data set where target data is located, and perform preprocessing on the multidimensional data set; a dimensionality reduction unit, used to sequentially perform feature extraction and data dimensionality reduction processing on the preprocessed multidimensional data set; a training unit, used to perform model training in a preselected classification model based on the dimensionality reduced data and extracted features to obtain a data recognition model; an identification unit, used to use the data to be identified as an input parameter of the data recognition model, perform model training, and determine whether the data to be identified is the target data based on the training results.
[0089] An embodiment of the present invention further provides a computer-readable storage medium, which stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned multidimensional data analysis method based on deep learning.
[0090] Those skilled in the art will understand that all or part of the steps in the method for implementing the above-mentioned embodiments can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including several instructions for making a single-chip microcomputer, a chip or a processor (processor) perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0091] The optional embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details in the above embodiments. Within the technical concept of the embodiments of the present invention, the technical scheme of the embodiments of the present invention can be subjected to a variety of simple modifications, and these simple modifications all belong to the protection scope of the embodiments of the present invention. It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the embodiments of the present invention will not further describe various possible combinations.
[0092] In addition, various embodiments of the present invention may be arbitrarily combined, and as long as they do not violate the concept of the embodiments of the present invention, they should also be regarded as the contents disclosed in the embodiments of the present invention.
Claims
1. A multidimensional data analysis method based on deep learning, characterized in that: The method comprises: Collecting the multidimensional data set where the target data is located, and performing preprocessing on the multidimensional data set; Performing feature extraction and data dimensionality reduction processing on the preprocessed multidimensional data set in sequence; Based on the reduced-dimensional data and extracted features, model training is performed in a pre-selected classification model to obtain a data recognition model; The data to be identified is used as an input parameter of the data identification model, model training is performed, and whether the data to be identified is target data is determined based on the training results.
2. The method according to claim 1, characterized in that The performing preprocessing on the multidimensional data set comprises: Data deduplication, missing value filling, outlier processing and data format conversion are sequentially performed on the multidimensional data set to obtain a preprocessed multidimensional data set.
3. The method according to claim 1, characterized in that: The processing rules for performing feature extraction on the preprocessed multidimensional data set are: Classify each item of data in the preprocessed multidimensional data set to obtain a classified data set corresponding to each data classification; wherein, The data categories include: High-dimensional data, speech signal data, image data, and time series data whose dimensions are greater than a preset dimensionality threshold; Based on the preset feature extraction algorithm corresponding to each data classification, parallel feature extraction is performed on each classification data set.
4. The method according to claim 3, characterized in that The preset feature extraction algorithm for high-dimensional data with a dimension greater than a preset dimension threshold is a principal component analysis algorithm; The feature extraction algorithm of the speech signal data is an independent component analysis algorithm; The feature extraction algorithm of the image data and time series data is a wavelet transform algorithm.
5. The method according to claim 3, characterized in that: After performing feature extraction on the preprocessed multidimensional data set, the method further comprises: Identify the most representative features, including: The most representative features are identified among the extracted features based on a filtering method or a wrapping method, and the most representative features identified are marked as key features.
6. The method according to claim 1, characterized in that Performing data dimensionality reduction processing on the preprocessed multidimensional data set includes: Based on the principal component analysis algorithm or the linear discriminant analysis algorithm, data dimension reduction processing is performed on the feature extraction or feature set to obtain a feature subset after dimension reduction; wherein, The feature extraction or feature set is subjected to data dimension reduction processing based on the principal component analysis algorithm, and the processing rules for obtaining the feature subset after dimension reduction are as follows: Perform standardization on each feature in the feature set and calculate the covariance matrix of each feature after standardization; Performing eigenvalue decomposition on the covariance matrix to obtain eigenvectors and corresponding eigenvalues; Sort the feature vectors according to the size of the corresponding eigenvalues, select the first K feature vectors, and construct the feature subset after dimensionality reduction.
7. The method according to claim 6, characterized in that Performing data dimension reduction processing on the feature extraction or feature set based on a linear discriminant analysis algorithm to obtain a feature subset after dimension reduction, including: Calculate the mean vector of each type of feature separately, and calculate the overall mean vector; Based on the mean vector of each feature and the overall mean vector, the intra-class scatter matrix and the inter-class scatter matrix are calculated, and the intra-class scatter matrix and the inter-class scatter matrix are combined to obtain a combined matrix; Performing eigenvalue decomposition on the combined matrix to obtain eigenvectors and eigenvalues; Sort the feature vectors according to the size of the corresponding eigenvalues, select the first K feature vectors, and construct the feature subset after dimensionality reduction.
8. The method according to claim 1, characterized in that: Based on the reduced-dimensional data and extracted features, model training is performed in the pre-selected classification model to obtain a data recognition model, including: Perform model training on the reduced-dimensional data and extracted features based on any model training algorithm including the gradient descent algorithm, the stochastic gradient descent algorithm, and the Adam algorithm to obtain an initial model; Based on the reserved dimensionality-reduced data and extracted features, the preset optimization targets of the initial model are optimized to obtain an optimized model as a data recognition model; wherein, The preset optimization goals include: Regularization, hyperparameter tuning, model fusion, data enhancement and network structure optimization.
9. A multidimensional data analysis system based on deep learning, characterized in that: The system comprises: A collection unit, used for collecting the multidimensional data set where the target data is located, and performing preprocessing on the multidimensional data set; A dimension reduction unit, used to perform feature extraction and data dimension reduction processing on the preprocessed multidimensional data set in sequence; A training unit, used for performing model training in a pre-selected classification model based on the dimension-reduced data and extracted features to obtain a data recognition model; The identification unit is used to use the data to be identified as an input parameter of the data identification model, perform model training, and determine whether the data to be identified is target data based on the training results.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the multidimensional data analysis method based on deep learning as described in any one of claims 1 to 8.
Citation Information
Cited By
Intelligent analysis system based on artificial intelligence
CN120578945A
