Big data analysis data preprocessing algorithm based on artificial intelligence neural network
Through deep neural networks combined with autoencoder and adaptive algorithms to optimize data preprocessing, the problems of long training time of high-dimensional data and difficulty in noise processing are solved, efficient and accurate data preprocessing is achieved, and the generalization ability and data analysis quality of the model are improved.
Patent Information
- Application Number
- CN202510436325.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-09
- Publication Date
- 2025-07-25
AI Technical Summary
The prior art has a long training time and is easy to overfit when processing high-dimensional data. The denoising effect of convolutional neural networks on complex noise is limited, and adaptive adjustment of standardization and normalization parameters is difficult.
The data preprocessing algorithm based on deep neural networks is adopted, combined with the autoencoder for unsupervised learning, and the strategy gradient method of adaptive algorithms and reinforcement learning is used to optimize the training process of neural networks through feature selection and dimensionality reduction, adaptive standardization and normalization technologies.
It significantly reduces the training time of high-dimensional data, suppresses the overfitting problem, enhances the processing ability of complex noise, and improves the generalization ability of the model and the efficiency and accuracy of data preprocessing.
Smart Images
Figure CN120372418A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of variable frequency control technology, and particularly to a data preprocessing algorithm for big data analysis based on an artificial intelligence neural network. Background Art
[0002] With the rapid development of big data technology, the data scale of various industries is increasing day by day, and the requirements for data analysis and processing have become increasingly complex. As an important step in data analysis, data preprocessing directly affects the effect of subsequent data mining and analysis. Traditional data preprocessing methods include data cleaning, missing value filling, outlier detection, data standardization, etc. However, with the continuous increase in the amount of data, traditional preprocessing methods often have problems such as low efficiency and poor effect. Especially when facing large-scale and high-dimensional data, traditional methods are difficult to handle complex features and noises.
[0003] In modern data analysis, artificial intelligence (AI) and deep learning technologies have been widely applied to the field of data preprocessing. Especially the introduction of neural networks has brought revolutionary changes to data processing. Neural networks, especially deep neural networks (DNNs) and convolutional neural networks (CNNs), can automatically learn the potential features of data from complex non-linear relationships and perform effective feature extraction and dimensionality reduction. This method is more efficient and accurate than traditional manual feature selection and dimensionality reduction methods. However, current deep learning methods still face some challenges in the data preprocessing process, which are mainly manifested in the following aspects.
[0004] The current deep learning methods face challenges in the data preprocessing process, mainly including: high-dimensional data processing: the training time for large-scale and high-dimensional data is long and it is easy to overfit; noise processing: the denoising ability of convolutional neural networks is limited and the effect is not good when dealing with complex noises; standardization and normalization: it is still difficult to adaptively adjust parameters to adapt to the characteristics of different data sets. Summary of the Invention
[0005] The main purpose of the present invention is to provide a data preprocessing algorithm for big data analysis based on an artificial intelligence neural network, which can effectively solve the problems of long training time and easy overfitting for processing high-dimensional data, limited denoising effect of convolutional neural networks on complex noises, and difficult adaptive adjustment of standardization and normalization parameters.
[0006] To achieve the above purpose, the technical solution adopted by the present invention is: a data preprocessing algorithm for big data analysis based on an artificial intelligence neural network, and the algorithm includes the following steps:
[0007] S1. Clean the original data to remove invalid data and outliers;
[0008] S2. Feature extraction and dimensionality reduction processing are performed on the data through a neural network to obtain a low-dimensional representation of the data.
[0009] Preferably, the structure of the neural network is a deep neural network, and unsupervised learning is performed using an autoencoder.
[0010] Preferably, the input layer of the neural network is the original data, the hidden layer is composed of multiple neurons, and the output layer is the data vector after dimensionality reduction.
[0011] Preferably, the training of the neural network uses a loss function to optimize the network parameters. The loss function is the mean squared error loss function (MSE), which is expressed as:
[0012]
[0013] where L is the loss function, that is, the target optimization function, which measures the difference between the model prediction result and the real data. n is the number of samples, representing the number of data points in the dataset. y i is the true value of the i-th sample, usually the label or target value in the dataset. is the predicted value of the i-th sample, representing the output value of the neural network model. This term represents the square of the prediction error of the i-th sample. The loss function penalizes the prediction deviation through the squared error.
[0014] Preferably, it further includes normalizing the data. The normalization formula is:
[0015]
[0016] where x ′ is the normalized data, representing the data value after normalization. x is the original data, representing the original input value in the dataset. μ is the mean of the data, representing the average value of all samples in the dataset. σ is the standard deviation of the data, representing the degree of dispersion between the sample values and the mean in the dataset.
[0017] Preferably, it further includes denoising the data using a neural network. A convolutional neural network (CNN) is used for noise identification and removal.
[0018] Preferably, the denoising operation calculates the difference between the original data and the denoised data, and uses a loss function to optimize the neural network weights. The loss function is the denoising loss function:
[0019]
[0020] where L denoising: The denoising loss function, which represents the loss function used to optimize the neural network and measures the error between the denoised data and the original data. n: The number of samples, i.e., the number of data points in the dataset. x i : The i-th sample value of the original data, representing the data with noise. : The i-th sample value after denoising, representing the data value after removing the noise.
[0021] Preferably, the parameters of each step in the preprocessing process are adjusted through an adaptive algorithm to further improve the quality of the data and the training effect of the neural network.
[0022] Preferably, the adaptive algorithm adopts the policy gradient method in reinforcement learning, and the formula is as follows:
[0023]
[0024] Where, The gradient of the loss function J with respect to the network parameter θ, representing the optimization objective, that is, adjusting the network parameters through the policy gradient method to improve the performance. Represents the expectation of the policy π θ (The expectation is based on the behavior under the current policy), that is, the weighted average of all possible state and action combinations. Represents the probability of taking action a in state s. This term is used to update the policy. a: Action, representing the decision-making choice in reinforcement learning, the behavior selected by the model based on the current policy. s: State, representing the current state of the environment, which is the input in reinforcement learning. R: Reward value, representing the feedback signal obtained after taking a certain action, usually the return of a certain state-action pair, used to measure the quality of this action.
[0025] Preferably, the algorithm realizes the efficient preprocessing of large-scale datasets by optimizing the training process of the neural network, and finally improves the accuracy and processing speed of data analysis.
[0026] Compared with the prior art, the present invention has the following beneficial effects:
[0027] 1. By introducing efficient feature selection and dimensionality reduction techniques, the present invention significantly reduces the training time of high-dimensional data, effectively suppresses the overfitting problem, can retain the key features of the data by optimizing the data representation method, reduces the computational complexity, improves the training efficiency of the model. In addition, a new regularization method is adopted to control the model complexity, further avoiding overfitting in high-dimensional data, thereby improving the generalization ability of the model.
[0028] 2. The present invention adopts an improved convolutional neural network architecture, combined with a multi-scale and adaptive denoising algorithm, effectively enhancing the processing ability of complex noise sources. By introducing a self-attention mechanism, the model can dynamically adjust the processing strategies for different types of noise, resulting in a significant improvement in the denoising effect. Especially when dealing with complex background noise, it can automatically distinguish noise from valid signals, thereby optimizing the data quality and improving the prediction accuracy.
[0029] 3. The present invention proposes an adaptive standardization and normalization technique that can automatically adjust parameters according to the specific characteristics of the dataset, reducing the complexity of manual adjustment. By combining the distribution information of the data, the model can dynamically adjust the standardization and normalization parameters during the training process, enabling more accurate preprocessing of different types of datasets. This technique improves the efficiency and flexibility of data preprocessing, reduces the dependence on manual intervention, and enhances the adaptability of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 is a flowchart of the present invention. DETAILED DESCRIPTION OF THE INVENTION
[0031] To make the technical means, creative features, achieved purposes, and functions of the present invention easy to understand, the present invention will be further described below in conjunction with specific embodiments.
[0032] Refer to Figure 1 shown, a big data analysis data preprocessing algorithm based on an artificial intelligence neural network, the algorithm includes the following steps:
[0033] S1. Clean the original data to remove invalid data and outliers;
[0034] S2. Extract features and perform dimensionality reduction processing on the data through a neural network to obtain a low-dimensional representation of the data.
[0035] By combining deep learning algorithms with an adaptive adjustment mechanism, efficient preprocessing of large datasets is achieved. First, an improved feature selection and dimensionality reduction method is adopted to reduce the training time of high-dimensional data and effectively avoid overfitting. Second, through an optimized convolutional neural network architecture and an adaptive denoising algorithm, the processing ability of complex noise is significantly improved. Finally, an adaptive standardization and normalization technique is introduced to automatically adjust parameters to adapt to the characteristics of different datasets, simplifying the preprocessing process;
[0036] This method can significantly improve the efficiency and accuracy of data preprocessing, reduce manual intervention, and improve the quality and reliability of subsequent analysis and mining
[0037] The structure of the neural network is a deep neural network, and an autoencoder is used for unsupervised learning.
[0038] Adopt a deep neural network structure, combine with an autoencoder for unsupervised learning, and automatically learn the low-dimensional representation of data through the autoencoder, thereby effectively reducing the redundant information of high-dimensional data, reducing the training time and improving the computational efficiency. The autoencoder can learn the latent features of data through the reconstruction process of the input data without labeled data, realizing efficient data dimensionality reduction and denoising processing;
[0039] This method not only improves the quality of data preprocessing, reduces the overfitting problem, but also can adaptively adjust the processing method of the data set, enhances the generalization ability of the model, and optimizes the overall performance of the data analysis and mining process.
[0040] The input layer of the neural network is the original data, the hidden layer is composed of multiple neurons, and the output layer is the data vector after dimensionality reduction.
[0041] Through the deep neural network structure, where the input layer receives the original data, and after being processed by the hidden layer of multiple neurons, the output layer generates the data vector after dimensionality reduction. The design of the autoencoder enables the network to automatically extract the main features of the data and perform effective dimensionality reduction during the unsupervised learning process. This process not only optimizes the data representation method, reduces redundant information, but also enhances the learning ability for complex data patterns through the multi-layer structure of the hidden layer neurons. Finally, the generated low-dimensional data vector not only retains the key information of the original data, but also reduces the risk of noise and overfitting, improving the efficiency and accuracy of data analysis and processing.
[0042] The training of the neural network uses a loss function to optimize the network parameters. The loss function is the mean squared error loss function (MSE), which is expressed as:
[0043]
[0044] where L is the loss function, that is, the target optimization function, which measures the difference between the model prediction result and the real data. n is the number of samples, representing the number of data points in the data set. y i is the real value of the i-th sample, usually the label or target value in the data set, is the predicted value of the i-th sample, representing the output value of the neural network model, This term represents the square of the prediction error of the i-th sample. The loss function penalizes the prediction deviation through the squared error.
[0045] The original data is received through the input layer of the deep neural network, undergoes non-linear feature transformation through the hidden layer of multiple neurons, and finally a data vector after dimensionality reduction is generated at the output layer. The structure of the autoencoder enables this network to learn the latent features of the input data without supervision and automatically achieve data dimensionality reduction;
[0046] In this way, the network can not only effectively remove redundant information, but also retain the core features of the data, thereby improving the training efficiency of the model and reducing the computational cost. At the same time, the dimension-reduced data vector is more compact, which can reduce the risk of overfitting, improve the generalization ability of the model, and make subsequent analysis or prediction more accurate and reliable.
[0047] It further includes standardizing the data, and the standardization formula is:
[0048]
[0049] where x ′ is the standardized data, representing the data value after standardization processing, x is the original data, representing the original input value in the dataset, μ is the mean of the data, representing the average value of all samples in the dataset, and σ is the standard deviation of the data, representing the degree of dispersion between the sample values and the mean in the dataset.
[0050] Receive the original data through the input layer in the deep neural network, perform complex non-linear mapping through the hidden layer of multiple neurons, and finally generate a dimension-reduced data vector in the output layer. In this process, the network uses the unsupervised learning method of the autoencoder to autonomously identify and extract the key information in the data and remove redundant data;
[0051] This method can significantly improve the efficiency of data processing, reduce the training time, and reduce the interference of data noise on the model by optimizing the dimension reduction process, improving the robustness and accuracy of the model. At the same time, the dimension-reduced data vector is more concise, easy for subsequent analysis and application, and helps to improve the overall processing performance.
[0052] It further includes using a neural network for denoising processing, and using a convolutional neural network (CNN) for noise identification and removal.
[0053] Through the deep neural network structure, the input layer receives the original data, and performs complex feature learning and transformation through the hidden layer of multiple neurons, and finally generates a dimension-reduced data vector in the output layer. In this process, the network uses the unsupervised learning method of the autoencoder to automatically extract the effective features of the data by minimizing the reconstruction error and complete the dimension reduction processing of the data;
[0054] This method not only reduces the redundant information of the input data, improves the data processing efficiency, but also can retain the key information, enhancing the generalization ability and accuracy of the model. Through the dimension-reduced data vector, subsequent analysis, classification or prediction can be better carried out, thereby optimizing the overall performance of data analysis and reducing the computational complexity and training cost.
[0055] The denoising operation calculates the difference between the original data and the denoised data, and uses a loss function to optimize the neural network weights. The loss function is the denoising loss function:
[0056]
[0057] where L denoising : the denoising loss function, representing the loss function used to optimize the neural network, measuring the error between the denoised data and the original data, n: the number of samples, the number of data points in the dataset, x i : the i-th sample value of the original data, representing the data with noise, the i-th sample value after denoising, representing the data value after removing the noise.
[0058] Through a deep neural network architecture, where the input layer receives the original data, and through the hidden layers with multiple neurons for feature extraction and learning, finally the output layer generates the data vector after dimensionality reduction. In this process, the neural network adopts the structure of an autoencoder, automatically identifying and extracting the key information in the data through unsupervised learning, and performing efficient dimensionality reduction processing;
[0059] This method can effectively remove redundant information, while retaining the important features in the original data, improving the training efficiency of the model, reducing the computational burden, and the data vector after dimensionality reduction is more concise, suitable for subsequent analysis, classification or prediction, helping to improve the accuracy of data processing, reducing overfitting and improving the generalization ability of the model, thereby enhancing the overall performance.
[0060] Adjust the parameters of each step in the preprocessing process through an adaptive algorithm to further improve the quality of the data and the training effect of the neural network.
[0061] Perform data processing through a deep neural network. The input layer receives the original data, and through multiple hidden layers for feature learning and transformation, finally the output layer generates the data vector after dimensionality reduction. The network adopts the unsupervised learning method of an autoencoder, which can automatically identify and extract the potential features in the data and complete effective dimensionality reduction processing;
[0062] This method can reduce redundant information and retain the key information of the data, significantly improving the data processing efficiency and reducing the computational burden. In addition, through the concise data vector generated after dimensionality reduction, it helps to improve the training accuracy and generalization ability of the model, reduce the risk of overfitting, optimize the accuracy of data analysis and prediction, thereby enhancing the overall performance and application effect.
[0063] The adaptive algorithm adopts the policy gradient method in reinforcement learning, and the formula is as follows:
[0064]
[0065] Among them, the gradient of the loss function J with respect to the network parameters θ, representing the optimization objective, that is, adjusting the network parameters through the policy gradient method to improve performance, represents the expected value of the policy π θ (the expectation is based on the behavior under the current policy), that is, the weighted average of all possible state and action combinations, represents the probability of taking action a in state s. This term is used to update the policy. a: action, representing the decision-making choice in reinforcement learning, the behavior selected by the model based on the current policy. s: state, representing the current state of the environment, which is the input in reinforcement learning. R: reward value, representing the feedback signal obtained after taking a certain action, usually the return of a certain state-action pair, used to measure the quality of this action.
[0066] The input data is processed through a deep neural network architecture. First, the original data is received through the input layer, and feature extraction and transformation are performed through multiple hidden layers. Finally, the output layer generates a reduced-dimensional data representation. This process utilizes the unsupervised learning mechanism of the autoencoder to automatically identify important features in the data by minimizing the reconstruction error and effectively reduce the dimension;
[0067] This method can remove redundant information while retaining the key information in the data, thereby improving the data processing efficiency and accuracy. The reduced-dimensional data is more concise, suitable for subsequent analysis, classification, or prediction, reducing the computational complexity, enhancing the robustness and generalization ability of the model, and improving the overall performance and accuracy.
[0068] The algorithm optimizes the training process of the neural network to achieve efficient preprocessing of large-scale data sets, and finally improves the accuracy and processing speed of data analysis.
[0069] The input data is efficiently processed through a deep neural network. First, the original data enters the network through the input layer, then feature learning and transformation are performed through multiple hidden layers, and finally the output layer generates a reduced-dimensional data representation. Using the unsupervised learning method of the autoencoder, the network automatically identifies and extracts the key features in the data, and completes the dimensionality reduction by minimizing the reconstruction error;
[0070] This method can effectively remove the redundant information in the data, retain the core features, improve the efficiency and accuracy of data processing. At the same time, the reduced-dimensional data is more concise, providing a more representative input for subsequent data analysis, classification, or prediction, reducing the consumption of computing resources, improving the training speed and generalization ability of the model, and thus optimizing the performance and effect of the overall system.
[0071] The foregoing has shown and described the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above-mentioned embodiments, and what is described in the above embodiments and the specification is only to illustrate the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements fall within the scope of the present invention claimed. The scope of the present invention claimed is defined by the appended claims and their equivalents.
Claims
1. A big data analysis data preprocessing algorithm based on an artificial intelligence neural network, characterized in that, The algorithm includes the following steps: S1. Clean the original data to remove invalid data and outliers; S2. Extract features and perform dimensionality reduction on the data through a neural network to obtain a low-dimensional representation of the data.
2. The data preprocessing algorithm for big data analysis based on an artificial intelligence neural network according to claim 1, characterized in that, The structure of the neural network is a deep neural network, and unsupervised learning is performed using an autoencoder.
3. A data preprocessing algorithm for big data analysis based on an artificial intelligence neural network according to claim 2, characterized in that, The input layer of the neural network is the original data, the hidden layer consists of multiple neurons, and the output layer is the data vector after dimensionality reduction.
4. A data preprocessing algorithm for big data analysis based on an artificial intelligence neural network according to claim 3, characterized in that, The training of the neural network uses a loss function to optimize the network parameters. The loss function is the mean squared error loss function (MSE), which is expressed as: Among them, L is the loss function, that is, the objective optimization function, which measures the difference between the model prediction result and the real data. n is the number of samples, representing the number of data points in the dataset, and y i is the true value of the i-th sample, usually the label or target value in the dataset, the predicted value of the i-th sample, representing the output value of the neural network model, This term represents the square of the prediction error of the i-th sample. The loss function penalizes the prediction deviation through the squared error.
5. A data preprocessing algorithm for big data analysis based on an artificial intelligence neural network according to claim 1, characterized in that, It further includes normalizing the data. The normalization formula is: where x ′ is the standardized data, representing the data value after standardization, x is the original data, representing the original input value in the dataset, μ is the mean of the data, representing the average of all samples in the dataset, and σ is the standard deviation of the data, representing the degree of dispersion between the sample values and the mean in the dataset.
6. A data preprocessing algorithm for big data analysis based on an artificial intelligence neural network according to claim 1, characterized in that, It further includes denoising the data using a neural network. A convolutional neural network (CNN) is used for noise identification and removal.
7. The data preprocessing algorithm for big data analysis based on an artificial intelligence neural network according to claim 6, characterized in that, The denoising operation calculates the difference between the original data and the denoised data, and uses a loss function to optimize the neural network weights. The loss function is the denoising loss function: Among them, L denoising : The denoising loss function, which represents the loss function used to optimize the neural network and measures the error between the denoised data and the original data. n: The number of samples, i.e., the number of data points in the dataset. x i : The i-th sample value of the original data, representing the data with noise. The i-th sample value after denoising, representing the data value after noise removal.
8. A data preprocessing algorithm for big data analysis based on an artificial intelligence neural network according to claim 1, characterized in that, The parameters of each step in the preprocessing process are adjusted through an adaptive algorithm to further improve the data quality and the training effect of the neural network.
9. A data preprocessing algorithm for big data analysis based on an artificial intelligence neural network according to claim 8, characterized in that, The adaptive algorithm uses the policy gradient method in reinforcement learning. The formula is as follows: Among them, The gradient of the loss function J with respect to the network parameter θ, which represents the optimization objective, that is, to adjust the network parameter through the policy gradient method to improve the performance. Denotes the expected value of the policy π θ (The expectation is based on the behavior under the current policy), that is, the weighted average of all possible state and action combinations. Denotes the probability of taking action a in state s. This term is used to update the policy. a: Action, which represents the decision-making choice in reinforcement learning, the behavior selected by the model based on the current policy. s: State, which represents the current state of the environment and is the input in reinforcement learning. R: Reward value, which represents the feedback signal obtained after taking a certain action, usually the return of a certain state-action pair, and is used to measure the quality of this action.
10. A data preprocessing algorithm for big data analysis based on an artificial intelligence neural network according to claim 1, characterized in that, The algorithm realizes efficient preprocessing of large-scale data sets by optimizing the training process of the neural network, and finally improves the accuracy and processing speed of data analysis.