Data analysis and prediction system based on deep learning

Abstract

This invention discloses a data analysis and prediction system based on deep learning, belonging to the fields of data analysis and artificial intelligence technology. The system includes a data acquisition module, a data preprocessing module, a feature extraction module, a deep learning prediction module, a result output module, a model optimization module, a data storage module, an anomaly monitoring module, a human-computer interaction module, and a permission management module. Through a unique multi-source data fusion preprocessing algorithm, an improved deep learning prediction model, and a dynamic optimization mechanism, it achieves accurate analysis and efficient prediction of multi-domain and multi-type data. Its prediction accuracy is improved by more than 15% compared to existing technologies, and the response latency is controlled within 500ms. It also possesses strong environmental adaptability and data security, and can be widely applied to data analysis and prediction scenarios in finance, industry, healthcare, and other fields.

Description

Technical Field

[0001] This invention belongs to the field of data analysis and artificial intelligence technology, specifically relating to a data analysis and prediction system based on deep learning. Background Technology

[0002] With the advent of the big data era, the amount of data generated in various fields has exploded. Data analysis and prediction, as the core means of mining data value and guiding decision-making, have been widely used. Currently, most existing data analysis and prediction systems employ traditional machine learning algorithms or conventional deep learning models, which have several limitations: First, they are poorly adaptable to multi-source heterogeneous data, unable to efficiently process mixed structured, semi-structured, and unstructured data. Furthermore, asynchronous data acquisition and low preprocessing accuracy can easily lead to data distortion, affecting subsequent analysis and prediction results. Second, feature extraction is not comprehensive enough, often employing shallow feature extraction methods that fail to extract deep, effective features from the data, and resulting in excessive redundant features, reducing model efficiency and prediction accuracy. Third, the prediction models have weak generalization capabilities; conventional models such as LSTM and CNN suffer from vanishing gradients and slow convergence, making them unsuitable for data prediction needs in different fields and scenarios, and exhibiting significant prediction lag, failing to meet real-time prediction requirements. Fourth, model optimization mechanisms are imperfect, often employing fixed-parameter optimization algorithms that cannot dynamically adjust model parameters in real-time based on prediction errors, leading to decreased prediction accuracy after long-term operation. Fifth, the systems lack security, stability, and ease of human-computer interaction, resulting in issues such as data leakage, undetected system failures, and inconvenient user operation.

[0003] Furthermore, existing data analysis and prediction systems largely suffer from severe homogenization in their core algorithms and architectures, lacking unique designs and failing to effectively circumvent the shortcomings of existing technologies. Consequently, their prediction accuracy, operational efficiency, and environmental adaptability cannot meet the demands of practical applications. Therefore, developing a deep learning-based data analysis and prediction system that can circumvent existing technologies, possesses unique algorithms and architectures, and offers high prediction accuracy, high operational efficiency, and strong security has become an urgent technical challenge. Summary of the Invention

[0004] The purpose of this invention is to overcome the technical defects of existing data analysis and prediction systems, such as poor adaptability, low prediction accuracy, weak generalization ability, insufficient security, and serious homogenization. This invention provides a data analysis and prediction system based on deep learning, which achieves efficient processing and accurate prediction of multi-source heterogeneous data through a unique modular architecture design and algorithm improvement, thereby improving the system's operating efficiency, stability, and security, and meeting the practical application needs of various fields.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a data analysis and prediction system based on deep learning, comprising a data acquisition module, a data preprocessing module, a feature extraction module, and a deep learning prediction module connected in sequence, and a result output module, a model optimization module, a data storage module, and an anomaly monitoring module respectively connected to the deep learning prediction module; further comprising a human-computer interaction module connected to both the data storage module and the result output module, and a permission management module connected to all the modules. The system achieves the acquisition, preprocessing, feature extraction, deep prediction, and result feedback of multi-source heterogeneous data through the collaborative work of its various modules. Each module adopts an independent operating architecture, enabling parallel processing. Furthermore, it avoids the homogenization defects of existing technologies through unique algorithm design. The specific workflow is as follows: The data acquisition module collects heterogeneous data from multiple sources and transmits it to the data preprocessing module. The data preprocessing module cleans and fuses the data before transmitting it to the feature extraction module. The feature extraction module extracts deep and effective features from the data and transmits them to the deep learning prediction module. The deep learning prediction module analyzes and predicts the feature data based on an improved deep learning model. The model optimization module dynamically optimizes the prediction model in real time. The anomaly monitoring module monitors the overall system operation status and the rationality of the prediction results. The data storage module stores all data and model parameters. The result output module outputs the prediction results. The human-computer interaction module enables user operation and information viewing. The permission management module controls the operation permissions of each user.

[0006] Furthermore, the data acquisition module adopts a multi-channel adaptive acquisition architecture, including at least three independent acquisition units, a signal conditioning unit, and a data synchronization unit; The acquisition unit can adapt to different types of data acquisition needs, including structured data (such as numerical and character data), semi-structured data (such as XML and JSON format data) and unstructured data (such as text, image, and audio data). Each acquisition unit has a built-in adaptive sampling rate adjustment module, and the sampling rate can be adjusted in real time according to the frequency of data change, with an adjustment range of 10Hz-1000Hz. The signal conditioning unit is used to filter, amplify, and perform analog-to-digital conversion on the acquired analog signal. The filtering adopts a second-order Butterworth low-pass filter algorithm, and its transfer function is:

[0007] in The cutoff frequency is 10Hz-500Hz, and the amplification factor can be adaptively adjusted according to the signal amplitude, with an adjustment range of 1-100 times. The data synchronization unit adopts a timestamp synchronization algorithm to align the data collected by different acquisition units in time, and the synchronization error is controlled within 10ms, ensuring the time sequence consistency of multi-source data and avoiding the problems of asynchronous acquisition and poor adaptability of multi-source data in the prior art.

[0008] Furthermore, the data preprocessing module includes a data purification unit, a data fusion unit, and a data standardization unit. The three units adopt a serial operation architecture, and each unit has a unique built-in processing algorithm to avoid the defects of low preprocessing accuracy and data distortion in the prior art. The data purification unit is used to remove outliers, missing values, and redundant data from the collected data. Outlier detection employs an improved Z-score algorithm, the calculation formula of which is:

[0009] in For the i-th data sample, The average value of the data sample. Let be the standard deviation of the data sample, and k be the adaptive adjustment coefficient, ranging from 1.8 to 2.5. When this occurs, the data is identified as an outlier and corrected using linear interpolation. The correction formula is as follows:

[0010] Missing value processing uses a neighborhood similarity-based interpolation algorithm, which performs accurate interpolation based on the distribution characteristics of the neighborhood data of the missing data. Redundant data removal uses an improved hash deduplication algorithm, which removes duplicate data by calculating and comparing the hash value of the data, achieving a deduplication accuracy of over 99.9%. The data fusion unit employs a weighted fusion algorithm to fuse data from different sources and of different types. The fusion formula is as follows:

[0011] in For data from the j-th source, Let J be the weight of the j-th source data, and satisfy the following condition: , The data is calculated adaptively based on its credibility; higher credibility results in greater weight. The data standardization unit employs an improved Min-Max standardization algorithm, mapping the fused data to the [0,1] interval. The standardization formula is as follows:

[0012] in The minimum value of the data sample. λ represents the maximum value of the data sample, and λ is an adjustment coefficient with a value range of 0.1-0.3, which avoids the problem of data distribution distortion in existing standardized algorithms.

[0013] Furthermore, the feature extraction module includes a shallow feature extraction unit, a deep feature extraction unit, and a feature filtering unit. It adopts a hierarchical extraction and precise filtering architecture, which can effectively extract effective features from the data and avoid the defects of incomplete feature extraction and excessive redundant features in the existing technology. The shallow feature extraction unit is used to extract the basic statistical features of the data, including mean, variance, range, skewness, and kurtosis, wherein the skewness calculation formula is:

[0014] The formula for calculating kurtosis is:

[0015] The deep feature extraction unit adopts an improved convolutional neural network (CNN) architecture, including 3 convolutional layers, 2 pooling layers, and 1 fully connected layer. The convolutional layers use an improved ReLU activation function, the expression of which is:

[0016] in It is an adaptive decay coefficient, with a value range of 0.01-0.05, which can effectively alleviate the gradient vanishing problem compared to the existing ReLU activation function; The pooling layer uses an adaptive max pooling algorithm, and the pooling kernel size can be adjusted in real time according to the feature map size, with an adjustment range of 2×2-4×4, to ensure the integrity of deep features; The feature selection unit employs an improved mutual information feature selection algorithm to calculate the mutual information value between each feature and the target variable. The mutual information calculation formula is as follows:

[0017] Where X is the feature variable, Y is the target variable, p(x,y) is the joint probability density of X and Y, and p(x) and p(y) are the marginal probability densities of X and Y, respectively. A mutual information threshold τ is set, with a value range of 0.3-0.5. Features with mutual information values ​​greater than τ are retained, and redundant features are removed. The feature dimension after filtering is reduced by more than 40% compared with the original deep feature dimension, and the feature recognition is improved by more than 30%.

[0018] Furthermore, the deep learning prediction module adopts a hybrid model architecture of improved LSTM (Long Short-Term Memory Network) and Transformer, including an input layer, an LSTM feature enhancement layer, a Transformer attention layer, a fusion layer and an output layer, which avoids the defects of weak generalization ability and prediction lag in existing prediction models. The input layer receives the effective features output by the feature extraction module and converts them into a vector form suitable for model processing, with a vector dimension of 64-256. The LSTM feature enhancement layer comprises two bidirectional LSTM units, each with 128-256 hidden layer neurons. It employs an improved forget gate, input gate, and output gate structure, with the forget gate's output formula being:

[0019] The output formula of the input gate is:

[0020] The cell state update formula is: ,

[0021] The output formula of the output gate is:

[0022] The hidden layer output formula is:

[0023] in It is the sigmoid activation function. These are the weight matrices for each gate. These are the bias terms for each gate. This is the output of the hidden layer from the previous time step. The input features at the current time, This represents the cell state at the previous time step; the Transformer attention layer employs an improved multi-head attention mechanism, with 4-8 heads, each head having 32-64 dimensions, and the attention weight calculation formula is as follows:

[0024] Where Q, K, and V are the query vector, key vector, and value vector, respectively. Let Q and K be the dimensions; the fusion layer uses a weighted summation method to fuse the output of the LSTM feature enhancement layer with the output of the Transformer attention layer, and the fusion formula is:

[0025] Where β is the fusion coefficient, with a value ranging from 0.4 to 0.6. The output of the LSTM feature enhancement layer. This is the output of the Transformer attention layer; The output layer uses a fully connected layer combined with either a sigmoid activation function (for classification prediction) or a linear activation function (for regression prediction) to output the prediction result. The probability output formula for classification prediction is as follows:

[0026] The output formula for regression prediction is:

[0027] Where W is the weight matrix of the fully connected layer, and b is the bias term.

[0028] Furthermore, the model optimization module includes an error calculation unit, a parameter adjustment unit, and a model update unit. It adopts a real-time dynamic optimization mechanism, which can adjust the model parameters in real time according to the prediction error, ensuring the prediction accuracy and generalization ability of the model, and avoiding the defects of lagging model optimization and unreasonable parameter adjustment in the prior art. The error calculation unit is used to calculate the error between the prediction result and the actual value. Different error calculation methods are used according to the prediction type. For classification prediction, an improved cross-entropy loss function is used, and the calculation formula is:

[0029] in Let be the actual label (0 or 1) of the i-th sample. Let γ be the predicted probability of the i-th sample, and let γ be the error adjustment coefficient, which ranges from 1.1 to 1.5 and can effectively improve the prediction accuracy of minority class samples. Regression prediction uses a modified mean squared error loss function, calculated as follows:

[0030] Where δ is the penalty coefficient, with a value ranging from 0.05 to 0.1, which imposes a heavier penalty on samples with larger prediction errors; The parameter adjustment unit employs an improved Adam optimization algorithm, and the learning rate can be adaptively adjusted. The learning rate update formula is as follows:

[0031] in This is the initial learning rate, ranging from 0.001 to 0.01. The decay coefficients are 0.9 and 0.999 respectively, and t is the number of iterations. Compared with the existing Adam optimization algorithm, it can effectively avoid learning rate oscillations and accelerate the convergence speed. The model update unit updates the model parameters of the deep learning prediction module in real time according to the optimized parameters output by the parameter adjustment unit. The update frequency can be adaptively adjusted according to the error changes, with an adjustment range of 1 time / 10 samples to 1 time / 50 samples, ensuring that the model is always in the optimal operating state.

[0032] Furthermore, the data storage module adopts a distributed storage architecture, including a primary storage node, a backup storage node, and a data backup node. It employs a unique data storage and backup strategy to avoid the shortcomings of existing technologies, such as low data storage security, untimely backup, and slow read speed. The main storage node is used to store real-time collected data, preprocessed data, feature data, prediction results and model parameters. It uses SSD solid-state drives for storage, with a read speed of over 2GB / s and a write speed of over 1GB / s. The backup storage node is synchronized with the primary storage node in real time. When the primary storage node fails, it can immediately switch to the backup storage node with a switching time controlled within 50ms, ensuring data continuity and availability. The data backup node employs an off-site backup strategy, performing a full backup of the primary storage node's data daily between 0:00 and 2:00 AM (low load period). Simultaneously, an incremental backup strategy is used, performing incremental backups of newly added data hourly. The compression ratio of the backup data is 4:1-6:1, and the backup data is encrypted using an improved AES encryption algorithm. The encryption formula is:

[0033] Where C is the encrypted data, P is the original data, and k is the encryption key. The key length is 256 bits to ensure the security of the backup data. The data storage module also has a built-in data lifecycle management unit, which sets different storage periods for different types of data. Expired data is automatically archived to release storage space. Archived data is compressed and stored with a compression rate of over 70%.

[0034] Furthermore, the anomaly monitoring module includes a system operation monitoring unit, a data anomaly monitoring unit, and a prediction result anomaly monitoring unit. It adopts a comprehensive, real-time monitoring mechanism, which can promptly detect anomalies in the system operation and data processing process, avoiding the shortcomings of incomplete anomaly monitoring and untimely response in the existing technology. The system operation monitoring unit is used to monitor the operating status of each module, including CPU utilization, memory usage, operating temperature and data transmission rate. It sets threshold ranges for each parameter: CPU utilization threshold is 85%, memory usage threshold is 80%, operating temperature threshold is 75℃, and data transmission rate threshold is 100Mbps. When a parameter exceeds the threshold, an early warning signal is immediately issued and abnormal information is recorded. The data anomaly monitoring unit is used to monitor the preprocessed data and feature data. It employs an improved isolated forest algorithm to calculate the anomaly score for each data sample. The anomaly score calculation formula is as follows:

[0035] Where n is the number of samples, E(h(x)) is the average path length of sample x in the isolated tree, and c(n) is the average path length when the number of samples is n. When the abnormal score is greater than 0.7, the data is determined to be abnormal data, the subsequent processing of the data is stopped immediately, and the data is fed back to the data preprocessing module for reprocessing. The prediction result anomaly monitoring unit is used to monitor the rationality of the prediction results and calculate the deviation rate between the prediction results and historical data. The deviation rate is calculated using the following formula:

[0036] in The deviation rate threshold is set to 20% based on the average of historical data. When the deviation rate exceeds the threshold, an early warning is issued immediately, and the model optimization module is triggered to perform emergency optimization of the prediction model. At the same time, the abnormal prediction results and their causes are recorded and archived.

[0037] Furthermore, the human-computer interaction module includes a display unit, an input unit, and a feedback unit. It adopts a simple and efficient interaction design to adapt to the operation needs of different users and avoid the shortcomings of inconvenient human-computer interaction and untimely feedback in the prior art. The display unit adopts a high-definition touch screen with a resolution of 1920×1080 pixels. It can display the system's operating status, data acquisition status, preprocessing results, feature extraction results, prediction results and abnormal information in real time. The display interface adopts a modular design, and users can customize the display content and display layout according to their needs. The input unit includes three methods: touch input, keyboard input, and voice input. The voice input uses an improved speech recognition algorithm with a recognition accuracy of over 98%, which can support users to complete operations such as data query, model parameter adjustment, and prediction task initiation through voice commands. The feedback unit is used to provide users with feedback on operation results and system status, including sound feedback, light feedback, and screen pop-up feedback. Sound feedback uses different frequencies of prompts to distinguish normal operation, warnings, and faults. Light feedback uses different colored LEDs to distinguish between normal operation, warnings, and faults: green indicates normal operation, yellow indicates warnings, and red indicates faults. Screen pop-up feedback is used to display detailed operation results and reasons for abnormalities, allowing users to quickly view and handle related issues. The human-computer interaction module also has a built-in help unit, providing detailed operation guides and answers to frequently asked questions to facilitate quick user operation.

[0038] Furthermore, the permission management module includes a user registration unit, a user authentication unit, and a permission allocation unit, and adopts a hierarchical permission management mechanism to ensure the security of the system and data, and avoid the shortcomings of the existing technology in that the permission management is not strict and there are security risks. The user registration unit is used for user registration. Users need to submit username, password, identity information, and their domain information. The password uses an improved encryption storage method, and the encryption formula is:

[0039] in The user's original password is used, with `salt` being a random salt value, 16 bits in length, and `hash` using the SHA-256 hash function to ensure password security. The user authentication unit is used for user login authentication, employing a two-factor authentication mechanism, including password authentication and dynamic verification code authentication. The dynamic verification code is generated using a time synchronization algorithm, with a valid duration of 60 seconds. The verification code generation formula is as follows:

[0040] Where time is the current timestamp, key is the authentication key, and mod is the modulo operation to ensure authentication security; The permission allocation unit divides users into three levels: administrator, operator, and viewer. Users at different levels have different operating permissions. Administrators have all operating permissions in the system, including user management, parameter setting, model optimization, data deletion, etc. Operators have permissions to collect data, preprocess it, start prediction tasks, and view results, but no permissions to modify parameters or manage users. Viewers only have permission to view prediction results and system operating status, and have no other operational permissions; The access control module records each user's operation log in real time, including operation time, operation content, operation result and IP address, which facilitates subsequent auditing and traceability.

[0041] This invention provides a data analysis and prediction system based on deep learning, which has the following beneficial effects: 1. Avoiding the homogenization problem of existing technologies: Each module of this invention adopts a unique architecture design and algorithm improvement, such as the improved Z-score outlier detection algorithm, the improved LSTM-Transformer hybrid prediction model, and the improved Adam optimization algorithm. These are fundamentally different from conventional algorithms and architectures in existing technologies, effectively avoiding the homogenization problem of existing technologies and possessing unique innovation.

[0042] 2. Strong adaptability to multi-source heterogeneous data: The data acquisition module adopts a multi-channel adaptive acquisition architecture, which can adapt to structured, semi-structured and unstructured data. The data preprocessing module adopts a unique fusion algorithm to achieve efficient fusion of multi-source data, solving the problems of poor adaptability of multi-source data and asynchronous acquisition in existing technologies.

[0043] 3. High prediction accuracy and fast response speed: By extracting deep and effective features through an improved feature extraction algorithm and removing redundant features, combined with an improved LSTM-Transformer hybrid prediction model and a real-time dynamic optimization mechanism, the prediction accuracy is improved by more than 15% compared with the existing technology, and the response latency is controlled within 500ms, which solves the problems of low prediction accuracy and obvious lag in the existing technology.

[0044] 4. High system stability and security: It adopts a distributed storage architecture, with fast master-slave node switching, a comprehensive backup strategy, and data encryption to ensure data security; the anomaly monitoring module provides comprehensive real-time monitoring to promptly detect and handle system and data anomalies; the access control module provides hierarchical management, records operation logs, avoids security risks, and solves the problems of insufficient system stability and security in existing technologies.

[0045] 5. Convenient human-computer interaction and wide adaptability: The human-computer interaction module supports multiple input methods, the display interface is modular and customizable, and a built-in help unit is convenient for different users to operate; the system can be widely used in data analysis and prediction scenarios in finance, industry, medical care and other fields, adapting to the needs of different fields and is highly practical. Attached Figure Description

[0046] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating the overall workflow of the system of the present invention; Figure 2 This is a flowchart of the data preprocessing module of the present invention; Figure 3 This is a flowchart of the deep learning prediction module of the present invention; Figure 4 This is a flowchart of the model optimization module of the present invention. Detailed Implementation

[0048] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses consistent with some aspects of this disclosure as detailed in the appended claims.

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0050] Example: This embodiment provides a data analysis and prediction system based on deep learning, applied to stock price data analysis and prediction scenarios in the financial field. Its specific structure and working process are as follows: 1. Data Acquisition Module: Employs three independent acquisition units to collect structured stock market data (such as numerical data like stock prices, trading volume, and turnover rate), semi-structured data (such as XML format data from listed company announcements), and unstructured data (such as financial news text data); the adaptive sampling rate adjustment module adjusts the sampling rate to 100Hz based on the frequency of stock price changes; the signal conditioning unit uses a second-order Butterworth low-pass filter algorithm with a cutoff frequency of... The frequency was set to 100Hz, and the acquired analog signal was filtered and the amplification factor was adjusted to 10 times. The data synchronization unit used a timestamp synchronization algorithm to time-align the data acquired by the three acquisition units, and the synchronization error was controlled within 10ms.

[0051] 2. Data Preprocessing Module: The data purification unit uses an improved Z-score algorithm with k set to 2.0 to detect outliers in stock price data and corrects them using linear interpolation; it also uses a neighborhood similarity-based interpolation algorithm to handle missing trading volume data; and an improved hash deduplication algorithm to remove duplicate financial news data, achieving a deduplication accuracy of 99.9%. The data fusion unit uses a weighted fusion algorithm, with a weight of 0.6 for core data such as stock price and trading volume, a weight of 0.2 for listed company announcements, and a weight of 0.2 for financial news data, satisfying the requirements of... The data standardization unit adopts the improved Min-Max standardization algorithm, with λ set to 0.2, to map the fused data to the [0,1] interval.

[0052] 3. Feature Extraction Module: The shallow feature extraction unit extracts the mean, variance, range, skewness, and kurtosis of the stock price data. Skewness S is calculated using the formula... The kurtosis K is calculated using the formula... The deep feature extraction unit adopts an improved CNN architecture, including 3 convolutional layers, 2 pooling layers and 1 fully connected layer. The α of the improved ReLU activation function is set to 0.03, and the pooling kernel size is adjusted to 3×3. The feature selection unit adopts an improved mutual information feature selection algorithm, with the mutual information threshold τ set to 0.4. Features with mutual information values ​​greater than 0.4 are retained, and the feature dimension after selection is reduced by 45% compared with the original deep feature dimension.

[0053] 4. Deep Learning Prediction Module: Employs a hybrid model architecture combining an improved LSTM and Transformer. The input layer converts the filtered features into 128-dimensional vectors. The LSTM feature enhancement layer includes two bidirectional LSTM units, and the hidden layer contains 192 neurons. The weight matrix utilizes an improved gate structure. The initial value is set to a random matrix, and the bias term is... The initial value is set to 0.1; the Transformer attention layer has 6 heads, each with 48 dimensions, and the attention weights are determined using the formula... Calculation; the β of the fusion layer is set to 0.5, and the outputs of the two layers are fused; the output layer uses a linear activation function, calculated using the formula... Output the stock price prediction results.

[0054] 5. Model Optimization Module: The error calculation unit uses an improved mean squared error loss function, with δ set to 0.08, to calculate the prediction error; the parameter tuning unit uses an improved Adam optimization algorithm with an initial learning rate of... Set it to 0.005. , Through formula The learning rate is adaptively adjusted; the model update unit updates the model parameters once every 30 samples based on the optimization parameters to ensure that the model is in the optimal running state.

[0055] 6. Data Storage Module: The primary storage node uses SSD solid-state drives with read speeds reaching 2.2GB / s and write speeds reaching 1.1GB / s; the backup storage node synchronizes with the primary storage node in real time, with failover time controlled within 45ms; the data backup node performs a full backup every day from 0:00 to 2:00, and an incremental backup every hour. The backup data compression ratio is 5:1, using an improved AES encryption algorithm with a key length of 256 bits, and is encrypted using the formula... Encryption; the data lifecycle management unit sets the real-time data storage cycle to 3 months and the historical data storage cycle to 1 year, and automatically archives expired data, achieving a compression rate of 75%.

[0056] 7. Anomaly Monitoring Module: The system operation monitoring unit monitors CPU usage, memory usage, operating temperature, and data transfer rate, with thresholds set at 85%, 80%, 75℃, and 100Mbps, respectively. A yellow warning is issued when the threshold is exceeded. The data anomaly monitoring unit uses an improved isolated forest algorithm. When the anomaly score is greater than 0.7, it is judged as abnormal data, subsequent processing is stopped, and feedback is provided. The prediction result anomaly monitoring unit calculates the deviation rate between the predicted stock price and the historical stock price average. The threshold is set at 20%. A red warning is issued when the threshold is exceeded, triggering the model optimization module to perform emergency optimization and recording the anomaly information.

[0057] 8. Human-Computer Interaction Module: The display unit adopts a 1920×1080 pixel high-definition touch screen, which modularly displays stock price data, preprocessing results, feature data, prediction results, and system operating status; the input unit supports touch, keyboard, and voice input, with a voice recognition accuracy rate of 98.5%; the feedback unit uses different frequency prompts, LED lights, and pop-up windows for feedback, with green LED lights indicating normal operation, yellow indicating warning, and red indicating fault; the built-in help unit provides detailed operation guides and answers to frequently asked questions.

[0058] 9. Access Control Module: The user registration unit requires users to submit their username, password, identity information, and department. Passwords are entered via a formula. Encrypted storage with a 16-bit salt length; the user authentication unit uses two-factor authentication, and the dynamic verification code is valid for 60 seconds, using a formula. The generation and permission allocation unit divides users into administrators, operators, and viewers. Administrators have full permissions, operators have permissions such as data collection and prediction startup, and viewers only have viewing permissions and record operation logs in real time.

[0059] In this embodiment, the stock price prediction accuracy of the deep learning-based data analysis and prediction system reaches over 88%, the response latency is controlled within 450ms, the system operates stably, and there are no data leaks or abnormal malfunctions. It effectively meets the needs of stock price data analysis and prediction in the financial field and has significant advantages compared with existing technologies.

[0060] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A data analysis and prediction system based on deep learning, characterized in that: It includes a data acquisition module, a data preprocessing module, a feature extraction module, and a deep learning prediction module that are electrically connected in sequence, as well as a result output module, a model optimization module, a data storage module, and an anomaly monitoring module that are electrically connected to the deep learning prediction module, and also includes a human-computer interaction module that is electrically connected to the data storage module and the result output module, and an access control module that is electrically connected to all the modules. The system achieves the acquisition, preprocessing, feature extraction, deep prediction, and result feedback of multi-source heterogeneous data through the collaborative work of its various modules. Each module adopts an independent operating architecture, enabling parallel processing. Furthermore, it avoids the homogenization defects of existing technologies through unique algorithm design. The specific workflow is as follows: The data acquisition module collects heterogeneous data from multiple sources and transmits it to the data preprocessing module. The data preprocessing module cleans and fuses the data before transmitting it to the feature extraction module. The feature extraction module extracts deep and effective features from the data and transmits them to the deep learning prediction module. The deep learning prediction module analyzes and predicts the feature data based on an improved deep learning model. The model optimization module dynamically optimizes the prediction model in real time. The anomaly monitoring module monitors the overall system operation status and the rationality of the prediction results. The data storage module stores all data and model parameters. The result output module outputs the prediction results. The human-computer interaction module enables user operation and information viewing. The permission management module controls the operation permissions of each user.

2. The data analysis and prediction system based on deep learning according to claim 1, characterized in that: The data acquisition module adopts a multi-channel adaptive acquisition architecture, including at least three independent acquisition units, a signal conditioning unit, and a data synchronization unit; The acquisition unit can adapt to different types of data acquisition needs, including structured data (such as numerical and character data), semi-structured data (such as XML and JSON format data) and unstructured data (such as text, image, and audio data). Each acquisition unit has a built-in adaptive sampling rate adjustment module, and the sampling rate can be adjusted in real time according to the frequency of data change, with an adjustment range of 10Hz-1000Hz. The signal conditioning unit is used to filter, amplify, and perform analog-to-digital conversion on the acquired analog signal. The filtering adopts a second-order Butterworth low-pass filter algorithm, and its transfer function is: in The cutoff frequency is 10Hz-500Hz, and the amplification factor can be adaptively adjusted according to the signal amplitude, with an adjustment range of 1-100 times. The data synchronization unit adopts a timestamp synchronization algorithm to align the data collected by different acquisition units in time, and the synchronization error is controlled within 10ms, ensuring the time sequence consistency of multi-source data and avoiding the problems of asynchronous acquisition and poor adaptability of multi-source data in the prior art.

3. The data analysis and prediction system based on deep learning according to claim 1, characterized in that: The data preprocessing module includes a data purification unit, a data fusion unit, and a data standardization unit. The three units adopt a serial operation architecture, and each unit has a unique built-in processing algorithm to avoid the defects of low preprocessing accuracy and data distortion in the existing technology. The data purification unit is used to remove outliers, missing values, and redundant data from the collected data. Outlier detection employs an improved Z-score algorithm, the calculation formula of which is: in For the i-th data sample, The average value of the data sample. Let be the standard deviation of the data sample, and k be the adaptive adjustment coefficient, ranging from 1.8 to 2.

5. When this occurs, the data is identified as an outlier and corrected using linear interpolation. The correction formula is as follows: Missing value processing uses a neighborhood similarity-based interpolation algorithm, which performs accurate interpolation based on the distribution characteristics of the neighborhood data of the missing data. Redundant data removal uses an improved hash deduplication algorithm, which removes duplicate data by calculating and comparing the hash value of the data, achieving a deduplication accuracy of over 99.9%. The data fusion unit employs a weighted fusion algorithm to fuse data from different sources and of different types. The fusion formula is as follows: in For data from the j-th source, Let J be the weight of the j-th source data, and satisfy the following condition: , The data is calculated adaptively based on its credibility; higher credibility results in greater weight. The data standardization unit employs an improved Min-Max standardization algorithm, mapping the fused data to the [0,1] interval. The standardization formula is as follows: in The minimum value of the data sample. λ represents the maximum value of the data sample, and λ is an adjustment coefficient with a value range of 0.1-0.3, which avoids the problem of data distribution distortion in existing standardized algorithms.

4. The data analysis and prediction system based on deep learning according to claim 1, characterized in that: The feature extraction module includes a shallow feature extraction unit, a deep feature extraction unit, and a feature filtering unit. It adopts a hierarchical extraction and precise filtering architecture, which can effectively extract effective features from the data and avoid the defects of incomplete feature extraction and excessive redundant features in the existing technology. The shallow feature extraction unit is used to extract the basic statistical features of the data, including mean, variance, range, skewness, and kurtosis, wherein the skewness calculation formula is: The formula for calculating kurtosis is: The deep feature extraction unit adopts an improved convolutional neural network (CNN) architecture, including 3 convolutional layers, 2 pooling layers, and 1 fully connected layer. The convolutional layers use an improved ReLU activation function, the expression of which is: in It is an adaptive decay coefficient, with a value range of 0.01-0.05, which can effectively alleviate the gradient vanishing problem compared to the existing ReLU activation function; The pooling layer uses an adaptive max pooling algorithm, and the pooling kernel size can be adjusted in real time according to the feature map size, with an adjustment range of 2×2-4×4, to ensure the integrity of deep features; The feature selection unit employs an improved mutual information feature selection algorithm to calculate the mutual information value between each feature and the target variable. The mutual information calculation formula is as follows: Where X is the feature variable, Y is the target variable, p(x,y) is the joint probability density of X and Y, and p(x) and p(y) are the marginal probability densities of X and Y, respectively. A mutual information threshold τ is set, with a value range of 0.3-0.

5. Features with mutual information values ​​greater than τ are retained, and redundant features are removed. The feature dimension after filtering is reduced by more than 40% compared with the original deep feature dimension, and the feature recognition is improved by more than 30%.

5. The data analysis and prediction system based on deep learning according to claim 1, characterized in that: The deep learning prediction module adopts a hybrid model architecture of improved LSTM (Long Short-Term Memory Network) and Transformer, including an input layer, an LSTM feature enhancement layer, a Transformer attention layer, a fusion layer and an output layer, which avoids the defects of weak generalization ability and prediction lag in existing prediction models. The input layer receives the effective features output by the feature extraction module and converts them into a vector form suitable for model processing, with a vector dimension of 64-256. The LSTM feature enhancement layer comprises two bidirectional LSTM units, each with 128-256 hidden layer neurons. It employs an improved forget gate, input gate, and output gate structure, with the forget gate's output formula being: The output formula of the input gate is: The cell state update formula is: ， The output formula of the output gate is: The hidden layer output formula is: in It is the sigmoid activation function. These are the weight matrices for each gate. These are the bias terms for each gate. This is the output of the hidden layer from the previous time step. The input features at the current time, This represents the cell state at the previous time step; the Transformer attention layer employs an improved multi-head attention mechanism, with 4-8 heads, each head having 32-64 dimensions, and the attention weight calculation formula is as follows: Where Q, K, and V are the query vector, key vector, and value vector, respectively. Let Q and K be the dimensions; the fusion layer uses a weighted summation method to fuse the output of the LSTM feature enhancement layer with the output of the Transformer attention layer, and the fusion formula is: Where β is the fusion coefficient, with a value ranging from 0.4 to 0.

6. The output of the LSTM feature enhancement layer. This is the output of the Transformer attention layer; The output layer uses a fully connected layer combined with either a sigmoid activation function (for classification prediction) or a linear activation function (for regression prediction) to output the prediction result. The probability output formula for classification prediction is as follows: The output formula for regression prediction is: Where W is the weight matrix of the fully connected layer, and b is the bias term.

6. The data analysis and prediction system based on deep learning according to claim 1, characterized in that: The model optimization module includes an error calculation unit, a parameter adjustment unit, and a model update unit. It adopts a real-time dynamic optimization mechanism, which can adjust the model parameters in real time according to the prediction error, ensuring the prediction accuracy and generalization ability of the model, and avoiding the defects of model optimization lag and unreasonable parameter adjustment in the existing technology. The error calculation unit is used to calculate the error between the prediction result and the actual value. Different error calculation methods are used according to the prediction type. For classification prediction, an improved cross-entropy loss function is used, and the calculation formula is: in Let be the actual label (0 or 1) of the i-th sample. Let γ be the predicted probability of the i-th sample, and let γ be the error adjustment coefficient, which ranges from 1.1 to 1.5 and can effectively improve the prediction accuracy of minority class samples. Regression prediction uses a modified mean squared error loss function, calculated as follows: Where δ is the penalty coefficient, with a value ranging from 0.05 to 0.1, which imposes a heavier penalty on samples with larger prediction errors; The parameter adjustment unit employs an improved Adam optimization algorithm, and the learning rate can be adaptively adjusted. The learning rate update formula is as follows: in This is the initial learning rate, ranging from 0.001 to 0.

01. The decay coefficients are 0.9 and 0.999 respectively, and t is the number of iterations. Compared with the existing Adam optimization algorithm, it can effectively avoid learning rate oscillations and accelerate the convergence speed. The model update unit updates the model parameters of the deep learning prediction module in real time according to the optimized parameters output by the parameter adjustment unit. The update frequency can be adaptively adjusted according to the error changes, with an adjustment range of 1 time / 10 samples to 1 time / 50 samples, ensuring that the model is always in the optimal operating state.

7. The data analysis and prediction system based on deep learning according to claim 1, characterized in that: The data storage module adopts a distributed storage architecture, including a primary storage node, a backup storage node, and a data backup node. It employs a unique data storage and backup strategy to avoid the shortcomings of existing technologies, such as low data storage security, untimely backup, and slow read speed. The main storage node is used to store real-time collected data, preprocessed data, feature data, prediction results and model parameters. It uses SSD solid-state drives for storage, with a read speed of over 2GB / s and a write speed of over 1GB / s. The backup storage node is synchronized with the primary storage node in real time. When the primary storage node fails, it can immediately switch to the backup storage node with a switching time controlled within 50ms, ensuring data continuity and availability. The data backup node employs an off-site backup strategy, performing a full backup of the primary storage node's data daily between 0:00 and 2:00 AM (low load period). Simultaneously, an incremental backup strategy is used, performing incremental backups of newly added data hourly. The compression ratio of the backup data is 4:1-6:1, and the backup data is encrypted using an improved AES encryption algorithm. The encryption formula is: Where C is the encrypted data, P is the original data, and k is the encryption key. The key length is 256 bits to ensure the security of the backup data. The data storage module also has a built-in data lifecycle management unit, which sets different storage periods for different types of data. Expired data is automatically archived to release storage space. Archived data is compressed and stored with a compression rate of over 70%.

8. The data analysis and prediction system based on deep learning according to claim 1, characterized in that: The anomaly monitoring module includes a system operation monitoring unit, a data anomaly monitoring unit, and a prediction result anomaly monitoring unit. It adopts a comprehensive, real-time monitoring mechanism, which can promptly detect anomalies in the system operation and data processing process, avoiding the shortcomings of incomplete anomaly monitoring and untimely response in the existing technology. The system operation monitoring unit is used to monitor the operating status of each module, including CPU utilization, memory usage, operating temperature and data transmission rate. It sets threshold ranges for each parameter: CPU utilization threshold is 85%, memory usage threshold is 80%, operating temperature threshold is 75℃, and data transmission rate threshold is 100Mbps. When a parameter exceeds the threshold, an early warning signal is immediately issued and abnormal information is recorded. The data anomaly monitoring unit is used to monitor the preprocessed data and feature data. It employs an improved isolated forest algorithm to calculate the anomaly score for each data sample. The anomaly score calculation formula is as follows: Where n is the number of samples, E(h(x)) is the average path length of sample x in the isolated tree, and c(n) is the average path length when the number of samples is n. When the abnormal score is greater than 0.7, the data is determined to be abnormal data, the subsequent processing of the data is stopped immediately, and the data is fed back to the data preprocessing module for reprocessing. The prediction result anomaly monitoring unit is used to monitor the rationality of the prediction results and calculate the deviation rate between the prediction results and historical data. The deviation rate is calculated using the following formula: in The deviation rate threshold is set to 20% based on the average of historical data. When the deviation rate exceeds the threshold, an early warning is issued immediately, and the model optimization module is triggered to perform emergency optimization of the prediction model. At the same time, the abnormal prediction results and their causes are recorded and archived.

9. The data analysis and prediction system based on deep learning according to claim 1, characterized in that: The human-computer interaction module includes a display unit, an input unit, and a feedback unit. It adopts a simple and efficient interaction design to adapt to the operation needs of different users and avoid the shortcomings of inconvenient human-computer interaction and untimely feedback in the existing technology. The display unit adopts a high-definition touch screen with a resolution of 1920×1080 pixels. It can display the system's operating status, data acquisition status, preprocessing results, feature extraction results, prediction results and abnormal information in real time. The display interface adopts a modular design, and users can customize the display content and display layout according to their needs. The input unit includes three methods: touch input, keyboard input, and voice input. The voice input uses an improved speech recognition algorithm with a recognition accuracy of over 98%, which can support users to complete operations such as data query, model parameter adjustment, and prediction task initiation through voice commands. The feedback unit is used to provide users with feedback on operation results and system status, including sound feedback, light feedback, and screen pop-up feedback. Sound feedback uses different frequencies of prompts to distinguish normal operation, warnings, and faults. Light feedback uses different colored LEDs to distinguish between normal operation, warnings, and faults: green indicates normal operation, yellow indicates warnings, and red indicates faults. Screen pop-up feedback is used to display detailed operation results and reasons for abnormalities, allowing users to quickly view and handle related issues. The human-computer interaction module also has a built-in help unit, providing detailed operation guides and answers to frequently asked questions to facilitate quick user operation.

10. The data analysis and prediction system based on deep learning according to claim 1, characterized in that: The permission management module includes a user registration unit, a user authentication unit, and a permission allocation unit. It adopts a hierarchical permission management mechanism to ensure the security of the system and data, and avoid the shortcomings of existing technologies such as lax permission management and potential security risks. The user registration unit is used for user registration. Users need to submit username, password, identity information, and their domain information. The password uses an improved encryption storage method, and the encryption formula is: in The user's original password is used, with `salt` being a random salt value, 16 bits in length, and `hash` using the SHA-256 hash function to ensure password security. The user authentication unit is used for user login authentication, employing a two-factor authentication mechanism, including password authentication and dynamic verification code authentication. The dynamic verification code is generated using a time synchronization algorithm, with a valid duration of 60 seconds. The verification code generation formula is as follows: Where time is the current timestamp, key is the authentication key, and mod is the modulo operation to ensure authentication security; The permission allocation unit divides users into three levels: administrator, operator, and viewer. Users at different levels have different operating permissions. Administrators have all operating permissions in the system, including user management, parameter setting, model optimization, data deletion, etc. Operators have permissions to collect data, preprocess it, start prediction tasks, and view results, but no permissions to modify parameters or manage users. Viewers only have permission to view prediction results and system operating status, and have no other operational permissions; The access control module records each user's operation log in real time, including operation time, operation content, operation result and IP address, which facilitates subsequent auditing and traceability.

Images