An internet information system integration service system

Abstract

The application discloses an internet information system integrated service system and belongs to the technical field of internet services; the system is used for solving the technical problems of low prediction accuracy and incapability of supporting dynamic scheduling caused by single model and insufficient data preprocessing in the prior art; through full-link collaborative design of multi-source data acquisition, singular spectrum analysis multi-scale decomposition, gradient boosting decision tree main model and bidirectional LSTM residual correction model cooperation and multi-step prediction, the defects of single feature dimension, insufficient data non-stationary processing, single model limitation and single prediction dimension in the prior art are solved, and the collaborative effect of all links makes the model prediction accuracy, real-time performance and decision scientificity significantly better than those of the traditional scheme, thereby providing reliable technical support for performance optimization of the internet information system integrated service.

Description

Technical Field

[0001] This invention relates to the field of Internet service technology, and more specifically to an Internet information system integration service system. Background Technology

[0002] Internet information system integration services are a comprehensive technical service that integrates hardware, network equipment, software applications, and related technical support to meet specific client needs. This service is typically used to create efficient and reliable information systems that support business processes and decision-making, including but not limited to requirements analysis, solution design, system development and customization, system integration, data migration, testing and optimization, training and support, maintenance and updates.

[0003] The operating environment of Internet systems is complex and ever-changing. Affected by factors such as network bandwidth fluctuations, uneven server loads, and surges in concurrent requests, performance data exhibits strong volatility, non-stationarity, and multi-timescale characteristics, such as sudden peaks, periodic fluctuations, and long-term trends. Existing Internet information system integration service solutions suffer from technical problems such as low prediction accuracy and inability to support dynamic scheduling due to single models and insufficient data preprocessing. Summary of the Invention

[0004] The purpose of this invention is to provide an Internet information system integration service system to solve the technical problems of low prediction accuracy and inability to support dynamic scheduling caused by the single model and insufficient data preprocessing in existing solutions.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] An internet information system integration service system, comprising:

[0007] Multi-source heterogeneous data acquisition and preprocessing module: Under different network environments, service loads and business types, it collects the operating parameters and service performance data of Internet information systems; it performs missing value imputation, outlier detection and normalization on the collected time series data to obtain the preprocessed performance data sequence S(t) and the corresponding set of operating parameters X(t).

[0008] Singular Spectrum Analysis Multiscale Decomposition Module: Input the performance data sequence S(t) into the singular spectrum analysis model, and extract multiscale subsequences containing trend component T(t), periodic component C(t) and noise component N(t) by constructing trajectory matrix, singular value decomposition and eigenvector grouping.

[0009] Gradient boosting decision tree main model training module: Using the running parameter set X(t) as input features, and the trend component T(t) and periodic component C(t) obtained from singular spectral analysis decomposition as joint output targets, the gradient boosting decision tree algorithm is used to train the main prediction model; the maximum depth of the tree, learning rate and subsampling ratio are adjusted through Bayesian optimization to minimize the prediction mean square error and obtain the prediction value of the main model for the trend-period components;

[0010] The bidirectional LSTM residual correction model training module calculates the residual sequence between the main model's predicted value and the true component. It takes the sliding time window of the residual sequence R(t) and the original performance data sequence S(t) as input to construct a bidirectional LSTM residual correction model, captures the contextual dependencies of the residuals, and outputs the multi-step predicted values ​​of the residuals.

[0011] Multi-step combined prediction model construction module: The trend-cycle prediction value of the main model is added to the residual multi-step prediction value of the bidirectional LSTM residual correction model, and combined with the characteristic parameters of the noise component N(t), a multi-step combined prediction model is constructed, and the performance prediction value of the next K steps is output.

[0012] Real-time running parameter analysis and output module: Input the real-time collected running parameters into the combined prediction model, output the preset performance index prediction results for the next K steps, and trigger dynamic resource scheduling and service quality warnings based on the output prediction results.

[0013] Preferably, different network environments include several levels of network bandwidth, network latency, and service load; the service types include file transfer and database query.

[0014] Preferably, the specific implementation steps of the singular spectrum analysis multi-scale decomposition module are as follows:

[0015] Constructing the trajectory matrix: For a performance data sequence S(t) of length N, select the embedding dimension L and construct an L×(N-L+1) trajectory matrix X = [S(1),S(2),...,S(N-L+1);S(2),S(3),...,S(N-L+2);...;S(L),S(L+1),...,S(N)];

[0016] Singular Value Decomposition: Performing singular value decomposition on the trajectory matrix X yields X = UΣV T Where U is the left singular matrix, Σ is the diagonal singular value matrix, and V is the right singular matrix;

[0017] Feature grouping and reconstruction: Based on the magnitude of the singular values, the feature vectors are divided into trend group, periodic group and noise group, and the trend component T(t), periodic component C(t) and noise component N(t) are reconstructed respectively.

[0018] Preferably, the hyperparameters of the decision tree master model are improved, with the hyperparameter optimization range being: maximum tree depth 3-8, learning rate 0.01-0.1, and subsampling ratio 0.6-0.9; the optimization objective is to minimize the MSE difference between the training set and the validation set.

[0019] The main model's prediction for the trend-cycle component is...

[0020] Preferably, the input to the bidirectional LSTM residual correction model is a time window of length L = 12, which includes the residual sequence R(t-L+1:t) and the original performance data S(t-L+1:t).

[0021] The training data for the bidirectional LSTM residual correction model is constructed using the sliding window method: for a residual sequence R(t) of length M, M-L+1 samples are generated, with the input of each sample being R(t-L+1:t) and S(t-L+1:t), and the output being R(t+1:t+K).

[0022] The residual sequence is

[0023] The multi-step prediction value of the residual is

[0024] Preferably, the performance prediction value is ε k The noisy random term follows the order N(μ,σ). 2 ).

[0025] Preferably, the noisy random term ε in the multi-step combined prediction model k The following parameters are determined by statistically analyzing historical data of the noise component N(t): The mean μ = E[N(t)] and variance σ are calculated. 2 =Var[N(t)], then ε k ∈N(μ,σ 2 ).

[0026] Preferably, the selection rule for the embedding dimension L is: L = min(N / 2, 100), where N is the length of the performance data sequence.

[0027] Preferably, system operating parameters are collected in real time through a Kafka message queue, and the real-time data is cleaned online using the Flink stream processing platform;

[0028] Preset performance metrics, including service throughput and error rate.

[0029] Compared to existing solutions, the beneficial effects achieved by this invention are:

[0030] This invention addresses the shortcomings of existing technologies, such as single feature dimension, insufficient handling of data non-stationarity, limitations of single model, and single prediction dimension, through a multi-source data acquisition, singular spectrum analysis multi-scale decomposition, gradient boosting decision tree master model and bidirectional LSTM residual correction model collaboration, and multi-step prediction end-link collaborative design. The synergistic effect of each link makes the model prediction accuracy, real-time performance and decision scientificity significantly better than traditional solutions, providing reliable technical support for the performance optimization of Internet information system integration services. Attached Figure Description

[0031] The invention will now be further described with reference to the accompanying drawings.

[0032] Figure 1 This is a block diagram of an Internet information system integration service system according to the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] like Figure 1 As shown, the present invention is an Internet information system integration service system, comprising:

[0035] Multi-source heterogeneous data acquisition and preprocessing module: Under different network environments, service loads and business types, it collects the operating parameters and service performance data of Internet information systems; it performs missing value imputation, outlier detection and normalization on the collected time series data to obtain the preprocessed performance data sequence S(t) and the corresponding set of operating parameters X(t).

[0036] The different network environments include several levels of network bandwidth, network latency, and service load; the service types include file transfer and database query.

[0037] Specifically, the network bandwidth is available in three tiers: 100Mbps, 500Mbps, and 1Gbps.

[0038] Network latency is available in three levels: 5ms (excellent), 20ms (average), and 50ms (poor).

[0039] Service load is defined as: low concurrency (≤1000 requests / second), medium concurrency (1000-5000 requests / second), and high concurrency (>5000 requests / second).

[0040] In addition, the following data was collected using Prometheus (server monitoring), OpenTelemetry (service tracking), and ELK logging system (network logs):

[0041] The set of operating parameters X(t) includes 7 dimensions: network bandwidth utilization (%), latency (ms), CPU utilization (%), memory utilization (%), disk I / O rate (MB / s), number of concurrent connections (number), and average response time (ms).

[0042] Performance data S(t): throughput (single / second), error rate (%), with throughput as the core predictive metric; t is the time index;

[0043] Among them, missing value imputation: cubic spline interpolation is used to imput consecutive missing values ​​less than 30 minutes.

[0044] Outlier detection: Based on the IQR method (interquartile range), points exceeding Q3+1.5IQR or below Q1-1.5IQR are marked as outliers and replaced with the mean of the preceding and following 10 minutes.

[0045] Normalization: Z-score standardization is used to scale the data to a distribution with a mean of 0 and a standard deviation of 1, so as to avoid the influence of the units on model training.

[0046] Through the above steps, we can obtain 30 consecutive days of preprocessed data, totaling 43,200 time steps, with t = 1 minute / step; of which, the training set accounts for 80%, with 34,560 data points, and the test set accounts for 20%, with 8,640 data points.

[0047] In this embodiment of the invention, by covering multiple scenarios such as network, load, and service type, it can be ensured that the data contains full-dimensional features of system operation, which can effectively improve the model's adaptability to complex environments; cubic spline interpolation can preserve the continuity of time series; IQR outlier detection can effectively filter out sudden noise, such as abnormal peaks caused by instantaneous server failures; Z-score standardization eliminates the differences in the units of different parameters, such as the difference in the numerical range of bandwidth utilization % and CPU utilization %, thus accelerating model convergence.

[0048] Singular Spectrum Analysis Multi-Scale Decomposition Module: The performance data sequence S(t) is input into the singular spectrum analysis model. Through constructing a trajectory matrix, singular value decomposition, and eigenvector grouping, a multi-scale subsequence containing a trend component T(t), a periodic component C(t), and a noise component N(t) is extracted. The specific implementation steps are as follows:

[0049] Constructing the trajectory matrix: For a performance data sequence S(t) of length N, select the embedding dimension L and construct an L×(N-L+1) trajectory matrix X = [S(1),S(2),...,S(N-L+1);S(2),S(3),...,S(N-L+2);...;S(L),S(L+1),...,S(N)];

[0050] Singular Value Decomposition (SVD): Performing SVD decomposition on the trajectory matrix X yields X = UΣV T Where U is the left singular matrix, Σ is the diagonal singular value matrix, and V is the right singular matrix;

[0051] Feature grouping and reconstruction: Based on the magnitude of the singular values, the feature vectors are divided into trend group, periodic group and noise group, and the trend component T(t), periodic component C(t) and noise component N(t) are reconstructed respectively.

[0052] Among them, the trend group consists of the first m large singular values;

[0053] Periodic group: the middle n moderately singular values;

[0054] Noise group: Residual small singular values;

[0055] Furthermore, the selection rule for the embedding dimension L is: L = min(N / 2, 100), where N is the length of the performance data sequence, ensuring that the rank of the trajectory matrix does not exceed L and that the computational efficiency is feasible;

[0056] In this embodiment of the invention, singular spectral analysis actively separates trends, periods, and noise in the data through SVD decomposition of the trajectory matrix, avoiding the mode mixing problem of traditional EMD algorithms. For example, high-frequency noise is mixed into periodic components, and the physical meaning of the components is clear: T(t) corresponds to the long-term trend, and C(t) corresponds to the business period. Multi-scale decomposition reduces the non-stationarity of the original data, providing a more stable input for subsequent models and effectively reducing their prediction error for periodic fluctuations. After decomposition, the standard deviation of each component is significantly reduced, and the non-stationarity of the data is weakened. Compared with traditional EMD decomposition, which requires more iterations to converge due to component mixing, this embodiment of the invention can effectively shorten the training time of the gradient boosting decision tree main model.

[0057] The gradient boosting decision tree main model training module uses the running parameter set X(t) as input features and the trend component T(t) and periodic component C(t) obtained from singular spectral analysis decomposition as joint output targets. It trains the main prediction model using the gradient boosting decision tree algorithm. Bayesian optimization is then used to adjust the maximum tree depth, learning rate, and subsampling ratio to minimize the mean squared error (MSE) of prediction, thus obtaining the main model's predictions for the trend-period components.

[0058] In this study, Bayesian optimization was used to adjust the gradient to improve the hyperparameters of the decision tree master model. The hyperparameter optimization range was: maximum tree depth 3-8, learning rate 0.01-0.1, and subsampling ratio 0.6-0.9. The optimization objective was to minimize the MSE difference between the training and validation sets, aiming to avoid overfitting. For example, the training set MSE was chosen to be 0.8σ. 2 Validation set MSE = 0.9σ 2 The parameter combination; σ is the standard deviation of the performance data;

[0059] In this embodiment of the invention, the gradient boosting decision tree gradually optimizes the prediction accuracy by iteratively fitting the residuals, so that each tree corrects the error of the previous tree. For high-dimensional nonlinear features, such as the quadratic relationship between CPU utilization and throughput, it can effectively improve the capture ability compared to random forest. By using Bayesian optimization, it is more efficient than grid search and can find a better combination of parameters.

[0060] Bidirectional LSTM residual correction model training module: Calculates the residual sequence between the main model's predicted values ​​and the true components. Using the sliding time window (window length L) of the residual sequence R(t) and the original performance data sequence S(t) as input, a bidirectional LSTM residual correction model is constructed to capture the contextual dependencies of the residuals and output the multi-step predicted values ​​of the residuals.

[0061] The bidirectional LSTM residual correction model takes a time window of length L = 12 as input, corresponding to 12 minutes of historical data, including the residual sequence R(t-L+1:t) and the original performance data S(t-L+1:t). The model structure is as follows: input layer (2L features) → bidirectional LSTM layer (128 neurons) → fully connected layer (64 neurons) → output layer (K ​​neurons, corresponding to residual predictions for the next K = 5 steps). The loss function is mean absolute error (MAE), the optimizer is Adam, and the learning rate is 0.001.

[0062] The training data for the bidirectional LSTM residual correction model is constructed using the sliding window method: for a residual sequence R(t) of length M, M-L+1 samples are generated, with the input of each sample being R(t-L+1:t) and S(t-L+1:t), and the output being R(t+1:t+K).

[0063] In this embodiment of the invention, bidirectional LSTM captures the "past→future" and "future→past" dependencies of the residuals simultaneously through forward and reverse time-series processing, thus correcting the information loss problem of unidirectional LSTM; the time window balances information retention and computational efficiency, and residual correction can effectively reduce the prediction error of the main model, laying the foundation for high-precision prediction of the combined model;

[0064] The gradient boosting decision tree master model handles interpretable deterministic errors, such as the direct correlation between running parameters and performance, while the bidirectional LSTM residual correction model handles uninterpretable temporal errors, such as residuals caused by random request fluctuations. This forms a hierarchical optimization architecture where the master model captures the main patterns and the correction model compensates for detailed errors, thus solving the overload defect of a single model having to learn both correlations and temporal sequences.

[0065] Multi-step combined prediction model building module: Combines the trend-cycle prediction values ​​of the main model Multi-step prediction of residuals compared to the bidirectional LSTM residual correction model By combining the summations and the characteristic parameters of the noise component N(t), a multi-step combined prediction model is constructed, outputting the performance prediction values ​​for the next K steps. ε k The noisy random term follows the order N(μ,σ). 2 );

[0066] The characteristic parameters are the mean μ and the variance σ. 2 ;

[0067] Noise random term ε in multi-step combined prediction model k The following parameters are determined by statistically analyzing historical data of the noise component N(t): The mean μ = E[N(t)] and variance σ are calculated. 2 =Var[N(t)], then ε k ∈N(μ,σ 2 ), used to quantify and predict uncertainty;

[0068] In this embodiment of the invention, the timeliness requirements of dynamic resource scheduling are met through multi-step prediction. The introduction of noise random terms quantifies the prediction uncertainty and provides a confidence reference for decision-making. For example, when the lower limit of the predicted value is lower than the threshold, the emphasis strategy is triggered.

[0069] The combined model integrates the feature association mining of gradient boosting decision trees with the temporal dependency capture advantages of bidirectional long short-term memory networks, forming a synergistic effect of "1+1>2", which can effectively reduce the prediction error of a single model.

[0070] Real-time operation parameter analysis and output module: Input the real-time collected operation parameters into the combined prediction model, output the preset performance index prediction results for the next K steps, and trigger dynamic resource scheduling and service quality warnings based on the output prediction results; K is a positive integer, and K=5 steps means the next 5 minutes.

[0071] Specifically, the system operating parameters X(t)′ are collected in real time through the Kafka message queue, and the real-time data is cleaned online using the Flink stream processing platform;

[0072] Preset performance metrics, including but not limited to service throughput and error rate;

[0073] The trigger condition for dynamic resource scheduling is: predicted throughput. When the capacity falls below a threshold, such as 80% of the current capacity, the system will automatically request elastic servers from the cloud platform to complete the expansion within 5 minutes.

[0074] The trigger condition for a service quality alert is: prediction error rate. When the request rate exceeds a threshold, such as 2%, the front-end system limits the request frequency, for example, by adjusting it from unlimited to 1 request / second.

[0075] It should be noted that with the Kafka+Flink stream processing architecture, the latency from data collection to model input is ≤2 seconds. Combined with multi-step prediction, sufficient time is reserved for resource scheduling. For example, if server expansion takes 5 minutes, the prediction results can trigger the operation in advance, avoiding service interruptions caused by traditional post-event remediation.

[0076] Furthermore, the dynamic scaling strategy based on prediction results enables server utilization to be dynamically adjusted from the traditional fixed configuration to on-demand scaling, reducing the cost of idle cloud resources.

[0077] In this embodiment of the invention, through multi-source data acquisition, singular spectrum analysis and multi-scale decomposition, gradient boosting decision tree master model and bidirectional LSTM residual correction model collaboration, and multi-step prediction full-link collaborative design, the shortcomings of existing technologies such as single feature dimension, insufficient handling of data non-stationarity, single model limitation, and single prediction dimension are specifically addressed. The synergistic effect of each link makes the model prediction accuracy, real-time performance and decision scientificity significantly better than traditional solutions, providing reliable technical support for the performance optimization of Internet information system integration services.

[0078] In the several embodiments provided by this invention, it should be understood that the disclosed system can be implemented in other ways. For example, the embodiments of the invention described above are merely illustrative; for example, the division of modules is only a logical functional division, and there may be other division methods in actual implementation.

[0079] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0080] Furthermore, the functional modules in the various embodiments of this invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or in a combination of hardware and software functional modules.

[0081] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the essential characteristics of the present invention.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. An Internet information system integration service system, characterized in that, include: Multi-source heterogeneous data acquisition and preprocessing module: Collects operating parameters and service performance data of Internet information systems under different network environments, service loads and business types; Missing value imputation, outlier detection and normalization are performed on the collected time series data to obtain the preprocessed performance data sequence S(t) and the corresponding set of running parameters X(t); Singular Spectrum Analysis Multiscale Decomposition Module: Input the performance data sequence S(t) into the singular spectrum analysis model, and extract multiscale subsequences containing trend component T(t), periodic component C(t) and noise component N(t) by constructing trajectory matrix, singular value decomposition and eigenvector grouping. Gradient boosting decision tree main model training module: The main prediction model is trained using the running parameter set X(t) as input features and the trend component T(t) and periodic component C(t) obtained by singular spectral analysis decomposition as joint output targets. The maximum depth of the tree, learning rate and subsampling ratio are adjusted by Bayesian optimization to minimize the prediction mean square error and obtain the prediction value of the main model for the trend-period components. The bidirectional LSTM residual correction model training module calculates the residual sequence between the main model's predicted value and the true component. It takes the sliding time window of the residual sequence R(t) and the original performance data sequence S(t) as input to construct a bidirectional LSTM residual correction model, captures the contextual dependencies of the residuals, and outputs the multi-step predicted values ​​of the residuals. Multi-step combined prediction model construction module: The trend-cycle prediction value of the main model is added to the residual multi-step prediction value of the bidirectional LSTM residual correction model, and combined with the characteristic parameters of the noise component N(t), a multi-step combined prediction model is constructed, and the performance prediction value of the next K steps is output. Real-time running parameter analysis and output module: Input the real-time collected running parameters into the combined prediction model, output the preset performance index prediction results for the next K steps, and trigger dynamic resource scheduling and service quality warnings based on the output prediction results.

2. The Internet information system integration service system according to claim 1, characterized in that, Different network environments include several levels of network bandwidth, network latency, and service load; business types include file transfer and database query.

3. The Internet information system integration service system according to claim 2, characterized in that, The specific implementation steps of the multi-scale decomposition module for singular spectrum analysis are as follows: Constructing the trajectory matrix: For a performance data sequence S(t) of length N, select the embedding dimension L and construct an L×(N-L+1) trajectory matrix X=[S(1),S(2),...,S(N−L+1);S(2),S(3),...,S(N−L+2);...;S(L),S(L+1),...,S(N)]; Singular Value Decomposition: Performing singular value decomposition on the trajectory matrix X yields X = UΣV T Where U is the left singular matrix, Σ is the diagonal singular value matrix, and V is the right singular matrix; Feature grouping and reconstruction: Based on the magnitude of the singular values, the feature vectors are divided into trend group, periodic group and noise group, and the trend component T(t), periodic component C(t) and noise component N(t) are reconstructed respectively.

4. The Internet information system integration service system according to claim 3, characterized in that, Improve the hyperparameters of the decision tree master model. The hyperparameter optimization range is: maximum tree depth 3-8, learning rate 0.01-0.1, and subsampling ratio 0.6-0.

9. The optimization objective is to minimize the difference in MSE between the training set and the validation set; The main model's prediction for the trend-cycle component is... .

5. The Internet information system integration service system according to claim 4, characterized in that, The input to the bidirectional LSTM residual correction model is a time window of length L=12, which includes the residual sequence R(t-L+1:t) and the original performance data S(t-L+1:t); The training data for the bidirectional LSTM residual correction model is constructed using the sliding window method: for a residual sequence R(t) of length M, M-L+1 samples are generated, with the input of each sample being R(t-L+1:t) and S(t-L+1:t), and the output being R(t+1:t+K). The residual sequence is ; The multi-step prediction value of the residual is , , ..., .

6. The Internet information system integration service system according to claim 5, characterized in that, Performance prediction value ; Let be a noisy random term, following the pattern N(μ, σ²), where μ is the mean and σ² is the variance.

7. The Internet information system integration service system according to claim 6, characterized in that, Noise random term in multi-step combined prediction model By statistically analyzing historical data of the noise component N(t), we determine: the mean of N(t) is μ=E[N(t)], and the variance is σ²=Var[N(t)]. ϵN(μ,σ²).

8. The Internet information system integration service system according to claim 3, characterized in that, The selection rule for the embedding dimension L is: L=min(N / 2, 100), where N is the length of the performance data sequence.

9. The Internet information system integration service system according to claim 1, characterized in that, The system's operating parameters are collected in real time using the Kafka message queue, and the real-time data is cleaned online using the Flink stream processing platform. Preset performance metrics, including service throughput and error rate.

Images