Digitally driven forward-looking mass data access method and system

Abstract

This invention belongs to the field of cloud computing and big data access scheduling technology, and discloses a data-driven, forward-looking method and system for massive data access. The method acquires node resource status data of access nodes and service traffic characteristic data of access tasks to form a global resource view; inputs historical service traffic characteristics into a hybrid time-series prediction model to obtain future service traffic prediction values; weights the future service traffic prediction values ​​with node resource status data to obtain a comprehensive load index; determines the migration-out node, the access tasks to be migrated, and the target node based on the comprehensive load index; synchronizes the task status of the access tasks to be migrated when the target node meets the acceptance conditions, and releases the corresponding task resources on the migration-out node after the target node takes over; and triggers expansion or contraction based on resource status. This invention can reduce scheduling lag and the risk of access task migration interruption.

Description

Technical Field

[0001] This invention relates to the field of cloud computing and big data processing technology, and in particular to a forward-looking resource scheduling, access task migration and elastic scaling method and system for massive business data access scenarios. Background Technology

[0002] As enterprise business systems, industrial equipment, log platforms, and multimedia platforms continue to connect to the cloud, the data access link needs to simultaneously handle multi-source heterogeneous data, sudden traffic surges, and cross-node task scheduling. Existing data access systems typically deploy acquisition agents, ETL tasks, or protocol access tasks on a fixed number of virtual machines, containers, or edge nodes. This deployment method is easy to maintain in low-fluctuation scenarios, but it exposes the following problems when business traffic changes rapidly.

[0003] 1. There is a static coupling between access tasks and node resources. When the data volume of a certain business line or protocol type increases, the CPU, memory, network bandwidth, or disk I / O of fixed access nodes may quickly approach their bottlenecks, while other access nodes remain under low load. If the existing system triggers scheduling only based on the current resource utilization, tasks are often migrated only after hotspots have formed, resulting in a lag in scheduling actions.

[0004] 2. Business traffic typically includes daily, weekly, and holiday fluctuations, as well as short-term spikes. When existing scheduling systems use a single prediction method, they are prone to slow response to short-term spikes or excessive sensitivity to short-term fluctuations, leading to frequent false triggers of scheduling when faced with complex traffic that has both long-term patterns and sudden data spikes. Prediction errors, once propagated to the scheduling side, can result in delayed capacity expansion, premature capacity reduction, or unnecessary migration of access tasks.

[0005] 3. Access tasks differ from ordinary offline tasks. Access tasks typically carry consumer locations, backlog locations, task priorities, and runtime node mappings. When using a migration method that stops the source task and restarts the target task, data duplication, data loss, or access interruption can easily occur within the migration window. For continuously written Kafka topics, protocol data streams, or real-time log streams, this issue will directly affect the integrity of downstream data lakes, data platforms, or real-time analytics systems.

[0006] 4. Most existing scheduling strategies rely on manually set thresholds or fixed weights. As business types, node specifications, and data peak-valley patterns change, fixed thresholds are difficult to adapt to in the long term. Without feedback and correction mechanisms after scheduling execution, the system struggles to adjust strategy parameters promptly based on throughput, migration latency, resource mismatch, and prediction errors.

[0007] Therefore, a forward-looking access solution is needed for massive data access scenarios, enabling the scheduling system to identify load risks before changes in business traffic and form an executable scheduling loop between access task migration and cloud resource scaling. Summary of the Invention

[0008] The purpose of this invention is to overcome the problems in the prior art, such as difficulty in timely scheduling based solely on the current load, data discontinuity caused by task migration, and poor long-term adaptability of fixed thresholds, and to provide a data-driven, forward-looking method and system for accessing massive amounts of data.

[0009] To achieve the above objectives, the technical solution provided by the present invention is as follows: Digital intelligence-driven, forward-looking methods for accessing massive amounts of data include: Obtain node resource status data of access nodes and service traffic characteristic data of access tasks, and form a global resource view; The historical service traffic characteristics in the global resource view are input into the hybrid time series prediction model to obtain the future service traffic prediction value. The future service traffic prediction value is then weighted with the node resource status data to obtain the comprehensive load index of the access node. The migration-out node, the access task to be migrated, and the target node are determined based on the comprehensive load index. If the target node meets the acceptance conditions, the task status of the task to be migrated is synchronized from the migration node to the target node, and the corresponding task resources on the migration node are released after the target node takes over the task to be migrated. The system will expand or shrink its capacity based on the resource status.

[0010] Preferably, the hybrid time series prediction model includes a first prediction model and a second prediction model. The first prediction model is used to extract long-term trend components and periodic peak-valley components from historical business traffic sequences. The second prediction model takes the residual sequence after removing the long-term trend components and the periodic peak-valley components as input and outputs a residual prediction result characterizing short-term sudden business traffic disturbances. The future business traffic prediction value is obtained by fusing the extrapolation result of the first prediction model and the residual prediction result.

[0011] Preferably, the first prediction model is a trend cycle decomposition model, and the second prediction model is a nonlinear sequence prediction model. The trend cycle decomposition model includes one of the Prophet model, ARIMA model, STL decomposition model, or exponential smoothing model, and the nonlinear sequence prediction model includes one of the LSTM neural network, GRU neural network, TCN network, or attention-based sequence prediction model.

[0012] Preferably, the comprehensive load index is obtained by weighting the CPU utilization rate, memory utilization rate and normalized future service traffic prediction value of the access node, and the sum of each weight coefficient is one.

[0013] Preferably, determining the migration node, the access task to be migrated, and the target node based on the comprehensive load index includes: identifying access nodes whose comprehensive load index is higher than a high load threshold as the migration node; estimating the traffic to be migrated based on the difference between the comprehensive load index of the migration node and the high load threshold; calculating a migration priority score based on the traffic, backlog, and task priority of the access task; and selecting the access task to be migrated based on the migration priority score.

[0014] Preferably, the target node is determined by: screening candidate nodes whose comprehensive load index is lower than the low load threshold from non-source access nodes; sorting the candidate nodes in ascending order of comprehensive load index, descending order of remaining CPU, and descending order of remaining memory; simulating the load increment of the candidate nodes after the migration of the access task to be migrated; and selecting the candidate node that minimizes the global comprehensive load variance as the target node when multiple candidate nodes meet the load constraints after migration.

[0015] Preferably, the task status includes at least one of consumption location, backlog data location, task running parameters, current processing batch identifier, and task priority, and after the target node takes over the access task to be migrated, it updates the global task mapping table and access node load information.

[0016] Preferably, after completing the migration, expansion, or reduction of the access task, the scheduling strategy parameters and / or the fusion weights of the hybrid time-series prediction model are updated based on the performance indicators after scheduling execution. The performance indicators include at least one of throughput improvement rate, migration latency, resource mismatch degree, and prediction error.

[0017] This invention also discloses a data-driven, forward-looking, massive data access system, comprising: The data acquisition module is used to acquire node resource status data of access nodes and service traffic characteristic data of access tasks, and form a global resource view; The hybrid prediction and analysis module is used to input the historical service traffic characteristics in the global resource view into the hybrid time series prediction model to obtain the future service traffic prediction value, and to weight the future service traffic prediction value with the node resource status data to obtain the comprehensive load index of the access node. The scheduling decision and migration module is used to determine the migration-out node, the task to be migrated and the target node based on the comprehensive load index. When the target node meets the acceptance conditions, the task status of the task to be migrated is synchronized from the migration-out node to the target node. After the target node takes over the task to be migrated, the corresponding task resources on the migration-out node are released. The elastic scaling control module is used to trigger expansion or contraction based on resource status.

[0018] Beneficial effects 1. Regarding prediction and scheduling timeliness, the global resource view integrates node resource status, task traffic, and task backlog status into the same data foundation, while the hybrid time-series prediction model further provides future business traffic forecasts. The comprehensive load index is jointly determined by the current resource status and future predicted traffic, enabling the system to identify load risks before hot nodes actually form, reducing the risk of delayed expansion or post-event migration caused by relying solely on current thresholds.

[0019] 2. Regarding resource utilization and target node selection, the comprehensive load index provides a unified judgment criterion for outgoing nodes, tasks to be migrated, and target nodes. Through post-migration load simulation and global comprehensive load variance arbitration, when multiple candidate nodes can accept the task, the target node is no longer randomly selected, but rather a target node that is conducive to cluster load balancing is prioritized.

[0020] 3. Regarding business continuity, the access task migration based on task status synchronization incorporates consumption points, backlog data locations, task execution parameters, target node takeover, and source node resource release into the same execution chain. For continuously accessing data streams, this chain reduces the risk of data duplication, data loss, and access interruption within the migration window.

[0021] 4. In terms of long-term operational adaptation, a further feedback update mechanism can be used to correct scheduling strategy parameters or predict fusion weights by utilizing throughput, migration latency, resource mismatch and prediction error, thereby reducing the maintenance cost of long-term reliance on manual threshold configuration. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating a preferred embodiment of the digital intelligence-driven forward-looking massive data access method provided by the present invention. Figure 2 This is a schematic diagram of data flow collaboration among modules in a digital intelligence-driven forward-looking massive data access system provided in a preferred embodiment of the present invention. Figure 3 This is a diagram illustrating the hybrid prediction mechanism architecture for extracting long-term time-series components and short-term nonlinear perturbation components in a preferred embodiment of the present invention. Detailed Implementation

[0023] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. The number of nodes, sampling period, threshold, and weight in the embodiments are used to illustrate the feasible path of the technical solution and do not constitute a limitation on the scope of protection. Unless the context otherwise requires, the access node can be a virtual machine, container, edge gateway, or physical server, and the access task can be a log collection task, a protocol data access task, an ETL task, or a streaming data writing task.

[0024] Example 1 like Figure 1 As shown in the figure, this embodiment provides a data-driven, forward-looking method for accessing massive amounts of data, which includes the following steps.

[0025] Step S1: Obtain the node resource status data of the access node and the service traffic characteristic data of the access task, and form a global resource view.

[0026] In this step, the system acquires node resource status data of the access nodes and service traffic characteristic data of the access tasks, and forms a global resource view. Node resource status data may include at least one of CPU utilization, memory utilization, network bandwidth usage, disk I / O, and number of connections. Service traffic characteristic data may include at least one of access data volume, current data backlog, historical service traffic sequence, and task priority.

[0027] In some preferred implementations, each access node deploys probes or data acquisition agents. The probes read CPU utilization, memory utilization, network bandwidth usage, disk I / O, and connection count from the cloud platform monitoring interface according to a preset sampling period, and write the sampling results to a distributed message queue. The data access agent synchronously reports the access data volume and current data backlog for each access task. The data acquisition module performs time alignment on node and task metrics near the same sampling time, and then aggregates them by access node to obtain a global resource view.

[0028] The global resource view can include a node resource status view and a task status view. The node resource status view records the node's unique identifier, timestamp, CPU utilization, memory utilization, bandwidth utilization, service traffic, and historical status sequences over multiple past time windows. The task status view records the task's unique identifier, the task's running node identifier, timestamp, amount of data accessed, current data backlog, task priority, and historical task status sequences over multiple past time windows.

[0029] In practical deployments, the global resource view can be stored in a real-time database, an in-memory database, or a time-series database. The real-time database only handles status queries and scheduling reads; the data content of the tasks themselves can still be written to the data lake, data platform, or business database. With this approach, the scheduling system can obtain the status information required for task migration without parsing the business data content.

[0030] Step S2: Input the historical service traffic characteristics in the global resource view into the hybrid time series prediction model to obtain the future service traffic prediction value, and weight the future service traffic prediction value with the node resource status data to obtain the comprehensive load index of the access node.

[0031] In this step, the system first obtains the predicted future business traffic based on the global resource view, and then combines the prediction results with node resource status data to form a comprehensive load index. The hybrid time-series prediction model includes a first prediction model and a second prediction model. The first prediction model is used to extract long-term trend components and periodic peak-valley components from historical business traffic sequences. The second prediction model takes the residual sequence after removing the long-term trend components and periodic peak-valley components as input and outputs the residual prediction results representing short-term sudden business traffic disturbances.

[0032] In some preferred implementations, the first prediction model uses the Prophet model, and the second prediction model uses an LSTM neural network. The system extracts historical service traffic sequences from multiple time windows for each access node from the global resource view. The Prophet model for Decompose to obtain the trend term Periodic terms and holiday items The residual sequence is obtained by the following formula: ; in, Represents the residual sequence. Represents a historical business traffic sequence. Indicates the trend term. Represents a periodic term. This indicates holiday periods. Residual sequences are used to express short-term disturbances, sudden hotspots, and nonlinear changes beyond long-term trends and periodic peaks and troughs.

[0033] Then the system will Constructing samples for supervised learning, based on past The residuals at each time step are used as input features, with future... The residuals at each time step are used as the prediction target to train an LSTM neural network. The LSTM neural network outputs the prediction result of the future residuals. Meanwhile, the Prophet model extrapolates the future. Trend items at each time step Periodic terms and holiday items .

[0034] The predicted future business traffic is obtained by combining the following formula: ; in, Indicates the future number Business traffic forecast values ​​for each time step. , and These represent the extrapolation results for the trend, cycle, and holiday items, respectively. This represents the residual prediction result output by the LSTM neural network. This indicates the maximum prediction step size corresponding to the preset time window in the future.

[0035] The combination of Prophet and LSTM described above is just one implementation method. In other implementations, the first prediction model can employ an ARIMA model, an STL decomposition model, or an exponential smoothing model to output the trend components and periodic peak-valley components in the historical traffic sequence; the second prediction model can employ a GRU neural network, a TCN network, or an attention-based sequence prediction model to predict short-term perturbations in the stripped residual sequence. As long as the first prediction model is responsible for extracting long-period or periodic components, and the second prediction model is responsible for predicting short-term nonlinear perturbations in the residual, a complete prediction model can be formed. Figure 3 The hybrid prediction mechanism is shown.

[0036] Access Node At any moment Comprehensive load index It can be calculated using the following formula: ; And satisfy: ;in, Indicates access node At any moment CPU utilization Indicates access node At any moment Memory usage. Indicates access node In the future Business traffic forecast values ​​for each time window. This indicates the maximum historical traffic of the cluster or the maximum theoretical processing capacity of a node. , and These represent the weighting coefficients for CPU utilization, memory utilization, and future predicted traffic, respectively.

[0037] and It can be adopted to Normalized values ​​within the interval. Divide by Then, use the same unit of measurement to avoid inconsistencies between the unit of business traffic and the unit of resource utilization. The weighting coefficient can be set according to the business's sensitivity to computing resources, memory resources, and future traffic changes. For example, in access scenarios with throughput priority and significant traffic peaks and troughs, a weighting coefficient can be used. , , This allows predicted traffic to account for a larger proportion of the overall load index; in scenarios where memory cache usage significantly impacts stability, it can improve... In computationally intensive protocol parsing scenarios, it can improve... .

[0038] Step S3: Determine the outgoing node, the access task to be migrated, and the target node based on the comprehensive load index.

[0039] In this step, the system determines the outgoing node, the access task to be migrated, and the target node based on the comprehensive load index. The purpose of this step is not simply to move data slices, but to select a suitable access task for migration and find a target node for that access task that can take over the continuous data flow.

[0040] The system iterates through all access nodes. If an access node... Comprehensive load index Above the high load threshold If so, the access node is determined as the migration node. The load reduction required for the migration node is estimated using the following formula: ; in, Indicates the node that migrates out The amount of load that needs to be reduced. Indicates access node The comprehensive load index, This indicates a high load threshold.

[0041] To convert the load difference into migrated service traffic, the system estimates the load-traffic sensitivity coefficient based on the overall load index of the migrated node and the current total traffic: ; in, Indicates the node that migrates out Load-flow sensitivity coefficient Indicates access node At any moment The total traffic. From this, the traffic to be migrated is obtained: ; in, Indicates the node that migrates out The estimated traffic to be migrated. This estimation enables the system to convert the load that needs to be reduced into task traffic that needs to be migrated.

[0042] At the outgoing node, the system calculates a migration priority score for each incoming task: ; in, Indicates access task Migration priority score. Indicates access task The amount of data or traffic accessed. This represents the maximum task traffic in the access task set. Indicates access task Data backlog This indicates the maximum backlog of tasks in the access task set. Indicates access task Task priority, , and These represent traffic weight, backlog weight, and priority penalty coefficient, respectively.

[0043] , and It can be configured according to business objectives and can meet them. When the system goal is to quickly relieve the throughput pressure on hot nodes, the following approach can be adopted: , , Prioritize migrating high-traffic tasks with significant backlogs; when the system goal is to reduce backlog risk, the following approach can be adopted. , , When the continuity of high-priority tasks is more important, efficiency can be improved. This makes high-priority tasks less likely to be selected as migration tasks due to penalties.

[0044] The system follows Sort the access tasks from highest to lowest, and add them to the migration list in sequence until the cumulative task traffic meets the requirements: ; in, This represents the sum of traffic for the selected tasks to be migrated. With this stopping condition, the traffic from the migrated tasks will be sufficient to cover the reduced traffic pressure on the migrated nodes.

[0045] The target node selection focuses on non-source access nodes. The system first filters nodes whose overall load index is below the low load threshold. The candidate nodes are then sorted in ascending order of overall load index, descending order of remaining CPU, and descending order of remaining memory. For each migration task, the system estimates the load increment of the candidate target nodes after migration. ; in, This represents the estimated load increase after the access task is migrated to the candidate target node. If the candidate target node... satisfy: ; Then the candidate target node meets the acceptance criteria. Among them, Indicates candidate target node At any moment The comprehensive load index.

[0046] If multiple candidate target nodes meet the admission criteria, the system further calculates the global comprehensive load variance after placing the access task to be migrated into each candidate node, and selects the candidate node that minimizes the global comprehensive load variance as the final target node. The global comprehensive load variance can be calculated using the following formula: ; in, Indicates the number of connected nodes. Indicates the access node after simulation migration The comprehensive load index, This represents the average global comprehensive load index after the simulated migration. This arbitration rule allows the target node selection to converge further from satisfying the constraints to satisfying the constraints and being more conducive to global equilibrium.

[0047] If none of the candidate target nodes meet the acceptance criteria, the system temporarily suspends the migration task or sends a scaling request to the elastic scaling control module. After the new access node is created and joins the cluster, it participates in the target node selection process again.

[0048] Step S4: If the target node meets the acceptance conditions, synchronize the task status of the task to be migrated from the migration node to the target node, and release the corresponding task resources on the migration node after the target node takes over the task to be migrated.

[0049] In this step, the system performs an access task migration based on task status synchronization. Task status can include at least one of the following: consumption position, backlog data location, task execution parameters, current batch identifier, and task priority. For Kafka topic access tasks, the consumption position can include the partition identifier and offset; for protocol data stream access tasks, the task status can include the connection session identifier, unacknowledged data segments, and write cursor.

[0050] At the start of the migration, the sending node synchronizes the task status of the incoming tasks to the target node. Upon receiving the task status, the target node starts the corresponding incoming task and continues processing data from the synchronized location or backlog position. During the migration, the sending node can continue processing incoming data until the target node has started up and is able to stably take over the tasks.

[0051] After the target node takes over, the system releases the corresponding task resources on the migrated node and updates the global task mapping table and the load information of the access node. Through state synchronization, takeover confirmation, and mapping update links, access tasks can maintain processing continuity within the migration window, reducing the risk of data duplication and data loss.

[0052] Step S5: Trigger expansion or reduction based on resource status.

[0053] In this step, the system triggers scaling up or down based on resource status. In a scaling up scenario, if the migration decision indicates insufficient overall resources, the elastic scaling control module calls the cloud resource interface to create a new access node. After the new access node completes initialization, data acquisition agent deployment, and heartbeat registration, it is added to the global resource view and participates in target node selection.

[0054] In scaling down scenarios, if certain access nodes remain below the low load threshold for an extended period and the access tasks running on them have been migrated to other nodes, the elastic scaling control module releases the access node. Before scaling down, it's confirmed that the access tasks on the node to be scaled down have been migrated out to avoid the resource release action affecting the continuous data flow.

[0055] Step S6: After completing the migration, expansion or reduction of the access task, update the scheduling strategy parameters and / or the fusion weight of the hybrid time-series prediction model according to the performance indicators feedback after scheduling execution. The performance indicators include at least one of throughput improvement rate, migration delay, resource mismatch degree and prediction error.

[0056] This step is a further implementation method. After completing the migration, expansion, or reduction of the access task, the system updates the scheduling policy parameters and / or the fusion weights of the hybrid time-series prediction model based on the performance indicators after scheduling execution. This step can be performed by a reinforcement learning model, an online optimizer, or a rule-based feedback controller.

[0057] In some preferred implementations, the system uses the current overall load index distribution, node resource status, prediction error, migration count, task backlog changes, and scaling results as state inputs. Action outputs include adjustments to high load thresholds, low load thresholds, overall load index weights, and prediction model fusion weights. Reward value. It can be constructed as follows: ; in, Indicates the throughput improvement rate. This represents the normalized migration delay. Indicates the degree of resource mismatch between nodes. This represents the normalized prediction error. , , and This is used as a reward weight. If the business prioritizes continuity, then increase the reward weight. If the business is more focused on resource costs, then increase... If the prediction bias has a greater impact on expansion and contraction, then increase... .

[0058] In actual deployment, the strategy feedback optimization module can update parameters periodically, or trigger updates after migration is complete, expansion is complete, or the prediction error exceeds a preset range. To avoid parameter oscillations, an upper limit can be set for each threshold or weight adjustment, for example, limiting the single adjustment range to 5% to 10% of the current value, and retaining the adjustment when performance indicators improve over multiple consecutive scheduling cycles.

[0059] Example 2 like Figure 2 As shown, this embodiment provides a data-driven, forward-looking massive data access system, including: a data acquisition module, used to acquire node resource status data of access nodes and service traffic characteristic data of access tasks, and form a global resource view; a hybrid prediction and analysis module, used to input the historical service traffic characteristics in the global resource view into a hybrid time-series prediction model to obtain future service traffic prediction values, and weight the future service traffic prediction values ​​with the node resource status data to obtain a comprehensive load index of the access nodes; a scheduling decision and migration module, used to determine the migration-out node, the access task to be migrated, and the target node according to the comprehensive load index, and when the target node meets the acceptance conditions, synchronize the task status of the access task to be migrated from the migration-out node to the target node, and release the corresponding task resources on the migration-out node after the target node takes over the access task to be migrated; and an elastic scaling control module, used to trigger expansion or contraction according to the resource status.

[0060] The data acquisition module operates for both access nodes and access tasks. This module reads CPU utilization, memory utilization, network bandwidth usage, disk I / O, and connection count from the access node side, and reads the amount of access data, current data backlog, task running node identifier, and task priority from the access task side. The acquired data is then aligned to timestamps and written to the global resource view.

[0061] The hybrid forecasting and analysis module reads historical service traffic sequences from the global resource view. Internally, this module may include a long-cycle component extraction unit, a residual disturbance prediction unit, and a load calculation unit. The long-cycle component extraction unit outputs trend components and periodic peak-valley components; the residual disturbance prediction unit outputs residual prediction results for short-term burst service traffic disturbances; and the load calculation unit synthesizes future service traffic forecasts with node resource status data into a comprehensive load index.

[0062] The scheduling decision and migration module uses the comprehensive load index as input. This module first identifies outgoing nodes with loads exceeding the high load threshold, then calculates the load reduction required for these outgoing nodes and the traffic to be migrated. Subsequently, it selects tasks to be migrated based on task traffic, backlog, and task priority, and filters target nodes from non-source nodes. If multiple candidate nodes meet the acceptance criteria, the module selects the final target node through global comprehensive load variance arbitration. During migration execution, the module coordinates the source node, target node, and global task mapping table, ensuring that the target node takes over the task before releasing the source node's resources.

[0063] The elastic scaling control module connects to the cloud resource interface or cluster management interface. During expansion, this module creates new access nodes and registers them to the global resource view; during scaling down, this module releases node resources after confirming that the access tasks on the node to be scaled down have been migrated out.

[0064] In further system implementations, the system may also include a policy feedback optimization module. This module receives performance metrics after scheduling execution and adjusts the high-load threshold, low-load threshold, comprehensive load index weight, and fusion weight of the hybrid time-series prediction model based on throughput, migration latency, resource mismatch, and prediction error. This module acts as an enhancement module, decoupled from the aforementioned main execution link. Even without this module deployed, the system can still complete forward-looking access task scheduling and state synchronization migration based on the comprehensive load index.

[0065] Experimental Example In a simulation scenario, 10 access nodes are deployed, each configured with a 4-core CPU, 16GB of memory, and 1Gbps network bandwidth. One control node runs the scheduling decision engine and policy feedback optimization module. Each access node runs a data acquisition agent, and access tasks include HTTP, SMTP, FTP, Telnet, POP3, and RIP protocol data access tasks. The access task processing chain involves accessing protocol data from a specified topic in Kafka and then writing it to the database. These tasks are distributed across the 10 access nodes for execution.

[0066] The simulated traffic includes weekday peak hours from 9:00 to 18:00, off-peak hours at night and on weekends, holiday effects, and burst traffic spikes lasting 5 minutes. The control scheme employs passive response scheduling based on CPU / memory thresholds, a single Prophet prediction model, a stop-restart migration method, and a horizontal scaling strategy based on CPU thresholds. The experimental scheme utilizes the comprehensive load index active scheduling, Prophet+LSTM hybrid time-series prediction, access task migration based on task state synchronization, active scaling based on service traffic prediction, and policy feedback optimization as described in this embodiment.

[0067] Key parameter configurations include: a data acquisition period of 1 minute, a prediction time window of the next 10 minutes, and a high load threshold. Low load threshold Comprehensive load index weight , , Task migration priority weight , , .

[0068] The prediction accuracy is compared in Table 1 below.

[0069] Table 1 Comparison of Prediction Accuracy The comparison between resource utilization and load balancing is shown in Table 2 below.

[0070] Table 2 Comparison of Resource Utilization Rate and Load Balancing The comparison between business continuity and migration quality is shown in Table 3 below.

[0071] Table 3 Comparison of Business Continuity and Migration Quality The comparison of elastic stretching response over time is shown in Table 4 below.

[0072] Table 4 Comparison of Elastic Strength and Response Aging The optimization effect of the strategy feedback is shown in Table 5 below.

[0073] Table 5 Strategy Feedback Optimization Effect Table The above results demonstrate that the hybrid prediction model reduces prediction errors in periodic, holiday, and burst traffic scenarios. Active scheduling based on the comprehensive load index improves the average resource utilization of the cluster, reduces load differences between nodes, and increases overall throughput. Access task migration based on task state synchronization reduces the number of interruptions during migration to zero and lowers average migration time and peak task backlog. Policy feedback optimization adjusts the high-load threshold and predicted traffic weights based on operational results, allowing the scheduling strategy to gradually align with the current business traffic patterns over 72 hours of operation.

[0074] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the present invention. Without departing from the concept of the present invention, the access node configuration, prediction model type, cloud resource interface, and policy feedback algorithm can all be replaced according to the specific deployment environment.

[0075] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for data access, characterized in that, include: Obtain node resource status data of access nodes and service traffic characteristic data of access tasks, and form a global resource view; The historical service traffic characteristics in the global resource view are input into the hybrid time series prediction model to obtain the future service traffic prediction value. The future service traffic prediction value is then weighted with the node resource status data to obtain the comprehensive load index of the access node. The migration-out node, the access task to be migrated, and the target node are determined based on the comprehensive load index. If the target node meets the acceptance conditions, the task status of the task to be migrated is synchronized from the migration node to the target node, and the corresponding task resources on the migration node are released after the target node takes over the task to be migrated. The system will expand or shrink its capacity based on the resource status.

2. The data-driven, forward-looking, massive data access method according to claim 1, characterized in that, The hybrid time series prediction model includes a first prediction model and a second prediction model. The first prediction model is used to extract long-term trend components and periodic peak-valley components from historical business traffic sequences. The second prediction model takes the residual sequence after removing the long-term trend components and periodic peak-valley components as input and outputs a residual prediction result that characterizes short-term sudden business traffic disturbances. The future business traffic prediction value is obtained by fusing the extrapolation result of the first prediction model and the residual prediction result.

3. The data-driven, forward-looking, massive data access method according to claim 2, characterized in that, The first prediction model is a trend cycle decomposition model, and the second prediction model is a nonlinear sequence prediction model. The trend cycle decomposition model includes one of the following: the Prophet model, the ARIMA model, the STL decomposition model, or the exponential smoothing model. The nonlinear sequence prediction model includes one of the following: the LSTM neural network, the GRU neural network, the TCN network, or the sequence prediction model based on the attention mechanism.

4. The data-driven, forward-looking, massive data access method according to claim 1, characterized in that, The comprehensive load index is obtained by weighting the CPU utilization rate, memory utilization rate and normalized future service traffic prediction value of the access node, and the sum of each weight coefficient is one.

5. The data-driven, forward-looking, massive data access method according to claim 1, characterized in that, The step of determining the migration node, the access task to be migrated, and the target node based on the comprehensive load index includes: identifying access nodes whose comprehensive load index is higher than the high load threshold as the migration node; estimating the traffic to be migrated based on the difference between the comprehensive load index of the migration node and the high load threshold; calculating a migration priority score based on the traffic, backlog, and task priority of the access task, and selecting the access task to be migrated based on the migration priority score.

6. The data-driven, forward-looking, massive data access method according to claim 5, characterized in that, The target node is determined by: screening candidate nodes whose comprehensive load index is lower than the low load threshold from non-source access nodes; sorting the candidate nodes in ascending order of comprehensive load index, descending order of remaining CPU, and descending order of remaining memory; simulating the load increment of the candidate nodes after the migration of the access task to be migrated; and selecting the candidate node that minimizes the global comprehensive load variance as the target node when multiple candidate nodes meet the load constraints after migration.

7. The data-driven, forward-looking, massive data access method according to claim 1, characterized in that, The task status includes at least one of consumption location, backlog data location, task running parameters, current processing batch identifier, and task priority, and after the target node takes over the task to be migrated, it updates the global task mapping table and the access node load information.

8. The data-driven, forward-looking, massive data access method according to claim 1, characterized in that, After completing the migration, expansion, or reduction of the access task, the scheduling strategy parameters and / or the fusion weights of the hybrid time-series prediction model are updated based on the performance indicators feedback after scheduling execution. The performance indicators include at least one of throughput improvement rate, migration latency, resource mismatch degree, and prediction error.

9. A data-driven, forward-looking, massive data access system, characterized in that: include: The data acquisition module is used to acquire node resource status data of access nodes and service traffic characteristic data of access tasks, and form a global resource view; The hybrid prediction and analysis module is used to input the historical service traffic characteristics in the global resource view into the hybrid time series prediction model to obtain the future service traffic prediction value, and to weight the future service traffic prediction value with the node resource status data to obtain the comprehensive load index of the access node. The scheduling decision and migration module is used to determine the migration-out node, the task to be migrated and the target node based on the comprehensive load index. When the target node meets the acceptance conditions, the task status of the task to be migrated is synchronized from the migration-out node to the target node. After the target node takes over the task to be migrated, the corresponding task resources on the migration-out node are released. The elastic scaling control module is used to trigger expansion or contraction based on resource status.

Images