Numerical prediction method based on deep neural network

Abstract

The invention discloses a numerical prediction method based on a deep neural network. The method comprises the following steps: S1, collecting and preprocessing training data; s2, constructing a lightweight neural network model comprising at least one attention mechanism module; s3, training the model by using the training data, minimizing a loss function, and taking a mean square error as the loss function; s4, performing knowledge distillation on the model; and S5, performing numerical prediction by using the smaller model, and through the lightweight neural network model, reducing the computing resource demand, and improving the speed and efficiency of model training. Through knowledge distillation, computing resources required during reasoning can be reduced, and prediction efficiency is improved. By integrating an attention mechanism, the model can highlight key features in input data, the interpretability of a model prediction result is improved, a data set is expanded through a data enhancement technology, and the generalization ability of the model is improved. And stable training of the model in the deep network is ensured through residual connection.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to a numerical prediction method based on a deep neural network. Background Art

[0002] With the rapid development of technology, image processing technology has become a core part of the field of computer vision. Traditional image processing methods, such as rule-based methods or simple machine learning algorithms, often fail to achieve ideal results when dealing with complex and variable image data. Problems such as details, textures, lighting changes, and occlusions in images pose great challenges to image analysis and understanding.

[0003] After retrieval, the invention patent with the Chinese patent number CN118246341A discloses a prediction method for the nonlinear stability bearing capacity of a single-layer reticulated shell structure with initial defects based on a graph neural network, including the following steps: determining initial global feature data according to design documents, and establishing a numerical model of a single-layer reticulated shell structure with initial defects; performing numerical simulation on the numerical model to obtain the nonlinear stability bearing capacity data of the single-layer reticulated shell; combining the initial global feature data and the nonlinear stability bearing capacity data to construct a data set; constructing a deep learning proxy model based on a graph neural network, and using the data set to train the model; using the trained model as the final proxy model for predicting the nonlinear stability bearing capacity of a single-layer reticulated shell structure with initial defects. Compared with the prior art, the invention patent with the Chinese patent number CN118246341A can accurately and quickly predict the nonlinear stability bearing capacity of an actual single-layer reticulated shell structure with initial defects, significantly reducing the computational cost of traditional methods and improving the efficiency and reliability of engineering design.

[0004] However, in the actual use process of the above method, the training data set is not diverse enough or not sufficient to cover all possible single-layer reticulated shell structure situations, resulting in weak generalization ability of the model; although the graph neural network is very powerful in dealing with graph structure data, its internal working principle is relatively complex and lacks intuitive interpretability; training a high-performance GNN model takes a long time. Therefore, a numerical prediction method based on a deep neural network is proposed. Summary of the Invention

[0005] The purpose of the present invention is to solve the deficiencies in the prior art that the training data set is not diverse enough or not sufficient to cover all possible single-layer reticulated shell structure situations, resulting in weak generalization ability of the model; although the graph neural network is very powerful in dealing with graph structure data, its internal working principle is relatively complex and lacks intuitive interpretability; training a high-performance GNN model takes a long time, and to propose a numerical prediction method based on a deep neural network.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A numerical prediction method based on a deep neural network, comprising the following steps:

[0008] 1. A numerical prediction method based on a deep neural network, characterized in that it comprises the following steps:

[0009] S1. Collect and preprocess training data, where the input vector includes but is not limited to time series data and financial indicator data, and the target vector is the corresponding prediction result. The preprocessing steps include data cleaning, normalization, and denoising. The data cleaning step includes removing outliers and missing values, and the normalization step normalizes the data set to the range [0, 1];

[0010] S2. Construct a lightweight neural network model containing at least one attention mechanism module. The model uses GraphSAGE as the basic architecture and includes residual connections to avoid the problem of gradient vanishing. Each residual block includes at least three convolutional layers and two skip connections;

[0011] S3. Use the training data to train the model, minimize the loss function, and use the mean squared error as the loss function. The training process includes forward propagation, backward propagation, and parameter update. The parameter update uses the Adam optimizer with a learning rate of 0.001;

[0012] S4. Perform knowledge distillation on the model to obtain a smaller model with similar prediction ability. The knowledge distillation process uses soft labels with a temperature parameter of 4, and trains the student model by minimizing the KL divergence between the output of the student model and the soft labels of the teacher model;

[0013] S5. Use the smaller model for numerical prediction. The prediction step includes inputting the preprocessed data and performing forward propagation through the student model to obtain the prediction result.

[0014] The above technical solutions further include:

[0015] Furthermore, the lightweight neural network model further has the following characteristics:

[0016] Use residual connections to avoid the problem of gradient vanishing, that is, add skip connections z after each layer l = x + f(x; W l ) where l represents the layer number, and this function is implemented using ResNet residual blocks;

[0017] Integrate the attention mechanism to improve the interpretability of the model. The attention score a i is calculated as

[0018]

[0019] Where α is a learned weight function, and this function is implemented using the self-attention mechanism Self-Attention;

[0020] Sparse matrix operations are adopted to reduce memory occupancy. For a sparse matrix A and a vector v, only non-zero elements are considered when calculating Av.

[0021] Furthermore, the knowledge distillation step further includes:

[0022] Using a well-trained large model f θ* As the teacher model, the Transformer model is adopted as the teacher model;

[0023] Transfer the soft labels of the teacher model To a smaller student model f φ , where T is the temperature parameter, which is set to four;

[0024] By minimizing the Kullback-Leibler divergence (KL(p ||p) between the output of the student model and the soft labels of the teacher model to train the student model. *

[0025] Furthermore, the attention mechanism module is configured to:

[0026] Automatically determine the important features in the input data, and the attention score a i Can be used to identify important features, such as identifying key economic indicators affecting stock prices in stock price prediction;

[0027] Provide feedback on which features are emphasized by the model, and explain the prediction results by visualizing the attention weights a i For example, in medical diagnosis, explain why the model focuses on a specific symptom.

[0028] Furthermore, the numerical prediction involves but is not limited to financial prediction, weather forecasting, or other time series-based data analysis, specifically such as predicting the closing price of a stock on a future day or the weather conditions within the next week.

[0029] Furthermore, the data preprocessing step further includes:

[0030] Clean the data: Remove outliers, missing values, etc., use the Z-score method to detect and remove outliers, and obtain the cleaned dataset D clean ;

[0031] ​​Data denoising: Apply noise filtering techniques, such as median filtering, to obtain the denoised dataset D denoised ;

[0032] Data standardization: Standardize the dataset D denoised to the range [0, 1], using min-max normalization, to obtain the standardized dataset D standardized ;

[0033] Data augmentation: Apply data augmentation techniques, such as random rotation, scaling, etc., to generate the extended dataset D augmented , where D augmented ={T(x i )|x i ∈D standardized}, T represents the data augmentation transformation, such as enhancing data by adding random wind speed changes in the weather forecast dataset.

[0034] Furthermore, the model training step further includes:

[0035] Initialize the model parameters θ, using the Xavier initialization method;

[0036] Define the loss function L(θ), using the mean squared error as the loss function;

[0037] Define the optimizer O, using the Adam optimizer;

[0038] Iteratively update the model parameters θ to minimize the loss function L(θ).

[0039] Furthermore, the model evaluation and optimization step further includes: Divide the validation set and the test set: Randomly select 20% of the data from the original data as the validation set and the test set;

[0040] Evaluate the model performance: Evaluate the model performance on the validation set, using the mean squared error as the evaluation metric;

[0041] Adjust the model structure and parameters: According to the evaluation results, adjust the model structure and parameters, such as increasing the number of hidden layers, changing the learning rate, etc.

[0042] Preferably, data security and privacy protection: Data encryption: Ensure that encryption technology is used during data collection, storage, and transmission to protect the confidentiality of the data. Privacy protection mechanism: For data containing sensitive personal information, implement differential privacy, anonymization, or pseudonymization processing to comply with data protection regulations. Permission management: Establish a strict data access permission management system to ensure that only authorized personnel can access and process the data.

[0043] The present invention has the following beneficial effects:

[0044] 1. In the present invention, through a lightweight neural network model: by using fewer layers or parameters such as GraphSAGE, the computational resource requirements are reduced, thereby improving the speed and efficiency of model training. Through knowledge distillation: by transferring the knowledge of a large model to a small model, the computational resources required during inference can be reduced, thereby improving the prediction efficiency. By integrating an attention mechanism, the model can highlight the key features in the input data, thereby providing a basis for the prediction and improving the interpretability of the model prediction results.

[0045] 2. In the present invention, through sparse matrix operations: for a sparse matrix A and a vector v, only non-zero elements are considered when calculating Av, significantly reducing the memory occupancy. By using data augmentation techniques to expand the dataset, the representativeness of the dataset is improved, thereby enhancing the generalization ability of the model. Through residual connections to avoid the problem of gradient vanishing and ensure stable training of the model in deep neural networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic structural diagram of a numerical prediction method based on a deep neural network proposed by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0047] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0048] As Figure 1 shown, a numerical prediction method based on a deep neural network proposed by the present invention includes the following steps:

[0049] 1. A numerical prediction method based on a deep neural network, characterized in that it includes the following steps:

[0050] S1. Collect and preprocess the training data, where the input vector includes but is not limited to time series data and financial indicator data, and the target vector is the corresponding prediction result. The preprocessing steps include data cleaning, normalization processing, and denoising. The data cleaning step includes removing outliers and missing values, and the normalization processing step normalizes the dataset to the range of [0, 1];

[0051] S2. Construct a lightweight neural network model including at least one attention mechanism module. The model uses GraphSAGE as the basic architecture and includes residual connections to avoid the problem of gradient vanishing, where each residual block includes at least three convolutional layers and two skip connections;

[0052] S3. Use the training data to train the model, minimize the loss function, and adopt the mean square error as the loss function. The training process includes forward propagation, backward propagation, and parameter update, where the Adam optimizer is used for parameter update and the learning rate is 0.001;

[0053] S4. Perform knowledge distillation on the model to obtain a smaller model with similar prediction ability. The temperature parameter of 4 is used for the soft labels in the knowledge distillation process, and the student model is trained by minimizing the KL divergence between the output of the student model and the soft labels of the teacher model;

[0054] S5. Use the smaller model for numerical prediction. The prediction steps include inputting the preprocessed data, performing forward propagation through the student model, and obtaining the prediction result.

[0055] In one embodiment, the lightweight neural network model further includes the following features:

[0056] Use residual connections to avoid the vanishing gradient problem, that is, add skip connections after each layer

[0057] z l = x + f(x; W l )

[0058] where l represents the layer number, and this function is implemented using the ResNet residual block;

[0059] Integrate the attention mechanism to improve the interpretability of the model. The attention score a i is calculated as:

[0060]

[0061] where α is a learned weight function, and the self-attention mechanism Self-Attention is used to implement it.

[0062] In one embodiment, the knowledge distillation step further includes:

[0063] Use a well-trained large model f θ* as the teacher model, and adopt the Transformer model as the teacher model;

[0064] Transfer the soft labels of the teacher model to a smaller student model f φ , where T is the temperature parameter, set to four;

[0065] By minimizing the KL divergence between the output of the student model and the soft labels of the teacher model The Kullback-Leibler divergence (KL(p* ||p) to train the student model.

[0066] In one embodiment, the attention mechanism module is configured to:

[0067] Automatically determine the important features in the input data, and the attention score a i can be used to identify the important features;

[0068] Provide feedback on which features are emphasized by the model, by visualizing the attention weights a i to explain the prediction results.

[0069] In one embodiment, the numerical prediction involves financial prediction, weather forecasting, or other time series-based data analysis.

[0070] In one embodiment, the data preprocessing step further includes:

[0071] Clean the data: Remove outliers, missing values, detect and remove outliers using the Z-score method, and obtain the cleaned dataset D clean ;

[0072] Data denoising: Apply noise median filtering to obtain the denoised dataset D denoised ;

[0073] Data standardization: Standardize the dataset D denoised to the range [0, 1], using min-max normalization, and obtain the standardized dataset D standardized ;

[0074] Data augmentation: Apply data augmentation techniques to generate the extended dataset D augmented , where D augmented = {T(x i ) | x i ∈ D standardized}, and T represents the data augmentation transformation.

[0075] In one embodiment, the model training step further includes:

[0076] Initialize the model parameters θ, using the Xavier initialization method;

[0077] Define the loss function L(θ), using the mean squared error as the loss function;

[0078] Define the optimizer O, using the Adam optimizer;

[0079] Iteratively update the model parameters θ through the training dataset D to minimize the loss function L(θ).

[0080] In one embodiment, the model evaluation and optimization step further includes: dividing the validation set and the test set: randomly extracting 20% of the data from the original data as the validation set and the test set;

[0081] Evaluating the model performance: evaluating the model performance on the validation set, and using the mean squared error as the evaluation metric;

[0082] Adjusting the model structure and parameters: adjusting the model structure and parameters according to the evaluation results.

[0083] In one embodiment, data collection and preprocessing: collecting relevant data sets, including input vectors and target vectors. Cleaning the data to remove outliers and missing values. Applying data denoising techniques, such as median filtering, to reduce the noise impact. Normalizing the data so that its values are between 0 and 1. Applying data augmentation techniques, such as random rotation, scaling, etc., to increase data diversity.

[0084] In one embodiment, constructing a lightweight neural network model: selecting a neural network architecture suitable for the task, such as CNN, RNN, etc. Adding an attention mechanism module to improve the model's attention to important features.

[0085] In one embodiment, using residual connections and ResNet residual blocks to solve the problem of gradient vanishing. Initializing the model parameters using Xavier. Model training: defining the mean squared error as the loss function to evaluate the accuracy of the model prediction. Selecting the Adam optimizer to adjust the model parameters. Using the training data set to iteratively update the model parameters and minimize the loss function.

[0086] In one embodiment, knowledge distillation: selecting a well-trained large model as the teacher model. Passing the soft labels of the teacher model to the student model. Training the student model by minimizing the Kullback-Leibler divergence between the output of the student model and the soft labels of the teacher model.

[0087] In one embodiment, model evaluation and optimization: randomly extracting 20% of the data from the original data as the validation set and the test set. Calculating the mean squared error on the validation set to evaluate the model performance, and adjusting the model structure and parameters according to the evaluation results to improve the model performance. Numerical prediction: using the trained model for numerical prediction. Outputting the prediction results.

[0088] Although the embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A numerical prediction method based on deep neural network, characterized in that: The following steps are involved: S1. Collect and preprocess training data, where the input vectors include but are not limited to time series data and financial indicator data, and the target vector is the corresponding prediction result. The preprocessing steps include data cleaning, standardization and denoising, where the data cleaning step includes removing outliers and missing values, and the standardization step standardizes the data set to the range of [0,1]; S2. Build a lightweight neural network model including at least one attention mechanism module, wherein the model uses GraphSAGE as the basic architecture and includes residual connections to avoid the gradient vanishing problem, wherein each residual block includes at least three convolutional layers and two skip connections; S3. Train the model using the training data to minimize the loss function, using mean square error as the loss function, and the training process includes forward propagation, back propagation and parameter update, wherein the parameter update uses the Adam optimizer with a learning rate of 0.001; S4. performing knowledge distillation on the model to obtain a smaller model with similar predictive power, wherein the knowledge distillation process uses soft labels with a temperature parameter of 4 to train the student model by minimizing the KL divergence between the student model output and the teacher model soft label; S5. Use the smaller model to perform numerical prediction, the prediction step comprising inputting the preprocessed data, performing forward propagation through the student model, and obtaining a prediction result.

2. The numerical prediction method based on deep neural network according to claim 1, characterized in that: The lightweight neural network model further includes the following features: Use residual connections to avoid the vanishing gradient problem, i.e. add skip connections after each layer z l =x+f(x;W l ) Where l represents the layer number, and ResNet residual block is used to implement this function; Integrate the attention mechanism to improve the interpretability of the model, the attention score a i Calculated as: Where α is a learned weight function, which is implemented using the self-attention mechanism.

3. The numerical prediction method based on deep neural network according to claim 1, characterized in that: The knowledge distillation step further includes: Use a large, fully trained model f θ* As the teacher model, the Transformer model is used as the teacher model; The soft label of the teacher model Passed to a smaller student model f φ , where T is the temperature parameter, set to four; By minimizing the student model output Soft label with teacher model The Kullback-Leibler divergence (KL(p * ||p) to train the student model.

4. The numerical prediction method based on deep neural network according to claim 1, characterized in that: The attention mechanism module is configured as follows: Automatically determine important features in the input data, the attention score a i Can be used to identify important features; Provide feedback about which features are valued by the model by visualizing the attention weights a i to explain the prediction results.

5. The numerical prediction method based on deep neural network according to claim 1, characterized in that: The numerical prediction involves financial prediction, weather forecast or other time series-based data analysis.

6. The numerical prediction method based on deep neural network according to claim 1, characterized in that: The data preprocessing step also includes: Clean the data: remove outliers and missing values, use the Z-score method to detect and remove outliers, and obtain the cleaned data set D clean ; Data denoising: Apply noise median filtering to obtain the denoised data set D denoised ; Data standardization: The dataset D denoised Standardize to the range of [0,1], use maximum and minimum normalization to get the standardized data set D standardized ; Data enhancement: Apply data enhancement techniques to generate an extended dataset D augmented , where D augmented ={T(x i )|x i ∈D standardized }, T represents the data enhancement transformation.

7. The numerical prediction method based on deep neural network according to claim 1, characterized in that: The model training step also includes: Initialize the model parameters θ using the Xavier initialization method; Define the loss function L(θ) and use the mean square error As a loss function; Define optimizer O and use Adam optimizer; The model parameters θ are iteratively updated through the training data set D to minimize the loss function L(θ).

8. The numerical prediction method based on deep neural network according to claim 1, characterized in that: The model evaluation and optimization step also includes: Divide the validation set and test set: Randomly extract 20% of the data from the original data as the validation set and test set; Evaluate model performance: Evaluate model performance on the validation set, using mean square error as the evaluation indicator; Adjust model structure and parameters: Adjust model structure and parameters based on the evaluation results.

Images