Adaptive data processing optimization method and device, equipment and medium

Abstract

The invention relates to the technical field of data processing, can be applied to business scenes such as financial science and technology and medical health, and discloses a self-adaptive data processing optimization method, device and equipment and a medium, and the method comprises the steps: collecting operation data, host performance data, network state data and historical task data of a target data source, and constructing an analysis model in combination with recovery parameters and strategy preference, predicting a task load state, resource consumption and execution duration, generating a task execution strategy, completing task scheduling and execution monitoring, and collecting execution feedback data for iterative optimization of the analysis model. According to the method, an analysis model is constructed by fusing multi-source system data and historical task information, a task execution strategy is generated in combination with a dynamic prediction result and a strategy weight, intelligent task scheduling and process monitoring are realized, and feedback data is used for model iterative optimization. The task execution efficiency is improved, the resource use rationalization is realized, and the model adaptive capability is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an adaptive data processing optimization method, device, equipment and storage medium. Background Art

[0002] In existing data backup systems, database administrators typically rely on pre-set, static policies for data protection. These policies typically include performing full backups on a fixed schedule, performing regular incremental backups between full backups, and performing log backups at set intervals. While this fixed policy execution model offered a certain degree of simplicity and controllability in its early stages, it has gradually become increasingly limited as business environments have become increasingly complex and data volumes and system resources have become more dynamic.

[0003] In the fintech business sector, data processing frequency and system usage peaks exhibit significant volatility. For example, during trading hours, at the end of a settlement period, or within the regulatory reporting window, system loads rise rapidly. Executing backup tasks based on fixed timeframes at these times is highly likely to conflict with business operations, leading to task queues or failures. In severe cases, this can even disrupt core trading operations. Furthermore, different systems have varying requirements for backup timeliness (Recovery Time Objective (RTO)) and integrity (Recovery Point Objective (RPO)). Traditional strategies cannot be dynamically adjusted as needed, making it difficult to balance the differences between high-frequency trading systems and low-frequency recording systems.

[0004] In the healthcare sector, hospital information systems (HIS), electronic medical records (EMR), and picture archiving and communication systems (PACS) have extremely high requirements for data continuity and real-time performance in their daily operations. For example, during emergency room visits or peak clinical hours, data writes are intensive and response times are stringent. Traditional backup mechanisms operating during these times can cause performance bottlenecks, impacting the physician experience or the success rate of writing critical information. Furthermore, the rate of change in medical data is irregular, and fixed-interval backup strategies are prone to data protection lags and resource waste, making them unable to meet the actual needs of rapid data recovery.

[0005] Furthermore, traditional backup host scheduling mechanisms are typically based on static regional divisions and task load thresholds, which hinders efficient utilization of system resources in cross-region deployments or hybrid cloud environments. Due to the lack of real-time scheduling capabilities, some hosts are often heavily loaded due to backlogged backup tasks, while others may be idle, leading to an imbalance in overall resource allocation and affecting the overall efficiency of backup tasks.

[0006] More critically, traditional database backup systems lack the ability to predict resource consumption trends, task execution durations, and potential conflict risks. When system resources are limited or the probability of task failure is high, operations personnel are often forced to manually intervene after a problem occurs, consuming significant time for troubleshooting and resetting policies. This increases labor costs and hinders adaptive optimization of policy execution. Summary of the Invention

[0007] The main purpose of the present invention is to provide an adaptive data processing optimization method, device, equipment and storage medium, aiming to solve the technical problems in the existing technology that the backup strategy is fixed, there is a lack of intelligent perception and predictive analysis of dynamic changes in data load and resource status, and it is impossible to achieve adaptive scheduling and optimization of backup tasks.

[0008] To achieve the above objectives, the present invention provides an adaptive data processing optimization method, comprising:

[0009] Obtaining operating data of a target data source, performance data of an execution host associated with the target data source, network status data, historical task data, recovery target parameters, and policy preferences;

[0010] Building and training a machine learning model based on the operation data, the performance data of the execution host, the network status data, and the historical task data to generate an analysis model;

[0011] By using the analysis model, the load status of the target data source in a future time period is analyzed to determine the task execution window, the data change rate of the target data source and the satisfaction of the recovery target parameters, as well as the resource consumption and execution time under different task types and parameter combinations;

[0012] Generate a task execution policy for the target data source based on the task execution window, data change rate, recovery target parameter satisfaction, resource consumption, execution time, and policy preference;

[0013] Dispatching the target task to the target execution host for execution according to the task execution strategy, and monitoring the execution status, resource consumption and execution duration of the target task;

[0014] The actual execution status, actual resource consumption, and actual execution duration of the target task are collected as new historical task data, and the new historical task data are fed back to the analysis model for iterative updating.

[0015] Furthermore, to achieve the above-mentioned object, the present invention provides an adaptive data processing optimization device, comprising:

[0016] A data acquisition module is used to obtain the operating data of the target data source, the performance data of the execution host associated with the target data source, network status data, historical task data, recovery target parameters and policy preferences;

[0017] A model training module is used to build and train a machine learning model based on the operation data, the performance data of the execution host, the network status data and the historical task data to generate an analysis model;

[0018] A prediction analysis module is used to analyze the load status of the target data source in a future time period through the analysis model to determine the task execution window, the data change rate of the target data source and the satisfaction of the recovery target parameters, as well as the resource consumption and execution time under different task types and parameter combinations;

[0019] A strategy generation module, configured to generate a task execution strategy for the target data source by comprehensively considering the task execution window, data change rate, recovery target parameter satisfaction, resource consumption, execution time, and strategy preference;

[0020] The task scheduling module is used to schedule the target task to the target execution host for execution according to the task execution strategy, and monitor the execution status, resource consumption and execution time of the target task;

[0021] The model updating module is used to collect the actual execution status, actual resource consumption and actual execution duration of the target task as new historical task data, and feed the new historical task data back to the analysis model for iterative updating.

[0022] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and an adaptive data processing optimization program stored in the memory and runnable on the processor, and when the adaptive data processing optimization program is executed by the processor, the steps of the adaptive data processing optimization method described above are implemented.

[0023] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which an adaptive data processing optimization program is stored, and when the adaptive data processing optimization program is executed by a processor, the steps of the adaptive data processing optimization method described above are implemented.

[0024] Beneficial effects: The present invention relates to the field of data processing technology and can be applied to business scenarios such as financial technology and medical health. It discloses an adaptive data processing optimization method, device, equipment and medium, including: obtaining the operating data of the target data source, the performance data of the associated execution host, network status data, historical task data, recovery target parameters and policy preferences, training a machine learning model to generate an analysis model, analyzing the load status, data change rate and recovery target parameter satisfaction in the future time period, and evaluating the resource consumption and execution time under different task types and parameter combinations, comprehensively analyzing the results and policy preferences to generate a task execution strategy, scheduling the target task to the target execution host and monitoring its execution status, resource consumption and execution time, collecting actual execution feedback as new historical task data for iterative updating of the analysis model. The present invention constructs an analysis model by fusing multi-source system data and historical task information, combines dynamic prediction results and policy weights to generate a task execution strategy and realize intelligent task scheduling and process monitoring, and then uses the feedback data for model iterative optimization to achieve improved task execution efficiency, rationalized resource use and enhanced model adaptability, effectively overcoming the problems of rigidity and low resource utilization of traditional static strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0026] Figure 1 A schematic diagram of an application environment of the adaptive data processing optimization method according to an embodiment of the present invention;

[0027] Figure 2 This is a flow chart of an embodiment of the adaptive data processing optimization method of the present invention;

[0028] Figure 3 Schematic diagram of functional modules of a preferred embodiment of the adaptive data processing and optimization device of the present invention;

[0029] Figure 4 A schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0030] Figure 5 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0032] The adaptive data processing optimization method provided by the embodiment of the present invention can be applied in the following situations: Figure 1In an application environment, the user terminal communicates with the server terminal through a network. The server terminal can obtain the operating data of the target data source, the performance data of the associated execution host, the network status data, the historical task data, the recovery target parameters and the policy preferences through the user terminal, train the machine learning model to generate an analysis model, analyze the load status, data change rate and recovery target parameter satisfaction in the future time period, and evaluate the resource consumption and execution time under different task types and parameter combinations, and generate a task execution strategy based on the comprehensive analysis results and the policy preferences. The target task is scheduled to the target execution host and its execution status, resource consumption and execution time are monitored, and the actual execution feedback is collected as new historical task data for iterative updating of the analysis model. The present invention constructs an analysis model by fusing multi-source system data and historical task information, combines dynamic prediction results and policy weights to generate a task execution strategy and realize intelligent task scheduling and process monitoring, and then uses the feedback data for model iterative optimization to achieve improved task execution efficiency, rationalized resource use and enhanced model adaptability, effectively overcoming the problems of rigidity and low resource utilization of traditional static strategies. Among them, the user terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server side can be implemented by an independent server or a server cluster composed of multiple servers. The present invention will be described in detail below through specific embodiments.

[0033] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of the adaptive data processing optimization method provided by the present invention. It should be noted that although a logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0034] like Figure 2 As shown, the adaptive data processing optimization method proposed by the present invention includes the following steps:

[0035] S10, acquiring operating data of a target data source, performance data of an execution host associated with the target data source, network status data, historical task data, recovery target parameters, and policy preferences;

[0036] In this embodiment, first, dynamic status information related to operation is collected from the target data source, including processor utilization, input and output load, transaction processing rate, and data change rate. Operation data is usually obtained through a real-time monitoring module deployed on the data source side. The monitoring module collects data by hooking kernel indicators or calling monitoring APIs. Processor utilization reflects the current computing pressure, input and output load indicates the disk or cache read and write density, transaction processing rate measures the business processing activity, and data change rate reflects the frequency of data set updates or insertions. The above indicators are statistically formed into a data set that can be used for modeling through time windows, which is time-series.

[0037] Secondly, it is necessary to collect performance status information for the execution host that is logically or physically bound to the target data source. This information primarily includes processor utilization, available memory capacity, storage device throughput, and network interface bandwidth utilization. This information is provided by a system monitoring agent deployed on the execution host. The system monitoring agent continuously collects resource usage information at the host level, using either a polling mechanism or an event-triggered mechanism. Processor utilization indicates CPU resource scheduling; available memory capacity is used to assess whether a task can be completed on the current host; storage device throughput is the data processing capacity of a disk or block device per unit time; and network interface bandwidth utilization measures the pressure on the communication channel.

[0038] In addition to operational and performance data, the network status between the target data source and the candidate execution hosts must also be captured. This data is obtained by sending probe packets to different hosts and calculating the return latency. This data is then used to analyze the average round-trip latency and peak available bandwidth of the network link. As a key determinant of data transmission and task scheduling latency, network status is implemented using standard TCP or UDP probing mechanisms. Effective bandwidth capacity can be inferred by combining packet loss rate and window retransmission mechanisms.

[0039] Furthermore, historical task records are required as training input for the prediction model. These records include fields such as the task type identifier, execution start and end timestamps, peak resource consumption, and the execution result status code. This data is stored in the task management database and is typically written in real time by the task orchestration system, reflecting the actual execution trajectory of the task. The task type identifier is used to cluster task characteristics, the start and end times are used to calculate duration, resource consumption data supports capacity prediction, and the status code reflects the stability of task execution.

[0040] Recovery target parameters are derived from user-defined constraints in the policy console. These primarily include the recovery time threshold and the recovery point threshold. The recovery time threshold defines the maximum timeframe for system recovery after an anomaly; the recovery point threshold corresponds to the maximum point in time at which data loss can be tolerated. Both are user-defined based on business continuity and disaster recovery requirements, parsed, and stored in the policy control component for subsequent scheduling logic reference.

[0041] Policy preferences define user preferences during task scheduling optimization, including cost control priorities, resource utilization limits, and task type priorities. This information is typically stored in a policy configuration file and entered through a visual interface to form configuration items. Configuration items include weight coefficients, priority values, or boundary limits, which are used to define constraints during the subsequent objective function optimization process.

[0042] In the financial industry, a database agent plug-in can be deployed to collect operational data from database nodes via a standard JMX interface, while also integrating with Prometheus or Zabbix systems to pull status information from the execution host. Network status is monitored using UDP probes deployed on backbone link nodes, combined with NetFlow traffic analysis to assess link availability. In the policy configuration interface, financial institution administrators can configure a recovery time limit of no more than 30 minutes for highly sensitive account databases, with a recovery point error limit of no more than 10 minutes. The policy file also sets the highest priority for funding tasks. The system parses and stores these policy preferences for model utilization.

[0043] In the healthcare sector, electronic medical record systems serve as data sources, using container-embedded monitoring plug-ins to collect processor stress and update rates. Edge node execution hosts send metrics to a unified management platform via agents. The hospital's internal network center periodically sends probe packets to the central host cluster to monitor link status and estimate load balancing. Historical task data is captured through a log system. Hospital information system administrators set recovery targets for critical systems within five minutes of sudden failures, with no more than one minute of medical record data loss. The policy prioritizes data integrity over cost optimization.

[0044] This embodiment establishes a decision-making foundation that supports predictive modeling and dynamic optimization by collecting multi-dimensional system operation data, task history information, and policy preferences. Operational data and performance indicators provide a reference for the current state, network status determines task reachability and latency risk, historical task records form learning samples, recovery objectives constrain task scheduling boundaries, and policy preferences provide optimization target weights, achieving the prerequisites for precise scheduling in dynamic environments. This approach overcomes the problems of rigid policy configuration, low resource utilization, and insufficient risk prediction capabilities inherent in traditional static scheduling models.

[0045] S20, building and training a machine learning model based on the operation data, the performance data of the execution host, the network status data, and the historical task data to generate an analysis model;

[0046] In this embodiment, the process of using operation data, performance data of the execution host, network status data and historical task data to build and train a machine learning model involves the analysis and fusion of multi-dimensional features. Operation data usually contains dynamic behavior information such as the write rate, query frequency, cache hit rate and transaction submission success rate of the data source. This information can reflect the current and short-term future load trend of the data source. The performance data of the execution host includes CPU utilization, memory usage, disk IO throughput and processing queue length, etc., which are used to characterize the hardware resource carrying capacity on which the task operation depends. Network status data reflects the data transmission capacity between different hosts, usually including average delay, packet loss rate, bandwidth utilization, etc., which directly affects the distribution efficiency and scheduling delay of the task. Historical task data provides a real operation record of the task, including task type, scheduling location, actual consumed resources, execution time and result status, which is used to model the performance differences of different tasks under different system conditions.

[0047] The above multi-class heterogeneous data is structured through feature engineering. This includes standardizing the time windows of each dimension feature, allowing different types of data to be modeled uniformly on the same timeline; constructing composite features, such as jointly encoding task type, average CPU utilization, and network latency into an input feature vector; and labeling historical task data, including resource consumption regression targets, success / failure classification labels, and execution time as regression targets.

[0048] The machine learning models used for training can be supervised learning models such as random forests, gradient boosting trees, and deep neural networks. Sequential modeling approaches such as temporal convolutional networks or recurrent neural networks can also be used to explore patterns in indicator changes over time. During training, sub-models for predicting resource consumption, task success probability, and task execution duration are constructed based on different prediction objectives. The training process includes model tuning steps such as stratified sampling, splitting the training and validation sets, adjusting the feature selection strategy, setting the loss function, and implementing an early stopping strategy.

[0049] After model training is completed, the above-mentioned multiple sub-models or integrated models are encapsulated into a unified analysis model. When receiving new operating data, execution host status and network status, the model can output load trends, resource consumption prediction values ​​and execution feasibility evaluation indicators in real time for subsequent task scheduling strategy derivation.

[0050] An implementation based on a lightweight distributed training architecture can be used to collect raw indicator data to a central training server and then construct a unified set of feature vectors. The LightGBM framework, based on a tree model, is selected to train a resource consumption prediction model, using historical task execution time as the regression target. A deep feedforward neural network is also used to construct a task success probability prediction model, with standardized load status and host resource indicators as input and a task completion probability score as output. For time-sensitive indicators, such as network status changes and system load fluctuations, long short-term memory networks (LSTMs) can be used for modeling to more accurately predict load status within future windows.

[0051] In another implementation, an unsupervised learning algorithm can be used to construct an anomaly detection module to assist the analytical model in detecting the risk level of executing a certain type of task under a given system state. For example, an anomaly distribution detection model can be trained using the isolation forest algorithm to score the degree of deviation between resource consumption and load, and the resulting score can be used as an input for scheduling policy evaluation.

[0052] For example, in the healthcare business, system load is lower at night, but certain data tables are frequently updated. Using a trained analytical model, the system predicts that certain tasks will have the lowest impact on storage bandwidth between 1:00 AM and 3:00 AM. Furthermore, the predicted completion time matches the recovery target. Therefore, the structural data backup task for the electronic medical record system is prioritized during this period to ensure data consistency and avoid resource conflicts.

[0053] In the fintech sector, high-frequency trading log processing systems must maintain a low RTO and schedule incremental backup tasks after peak weekday trading. An analytical model leverages historical task data and current host performance to predict peak resource consumption and scheduling feasibility. Within 30 minutes of a trade closing, the task is initiated on an execution node with minimal network latency, ensuring a dynamic balance between backup efficiency and resource stability.

[0054] This embodiment uses operational data, host status, network conditions, and historical task behavior data as input, and the trained analysis model has the ability to predict future resource consumption, execution success rate, and task duration. This method can simulate and evaluate different tasks and scheduling schemes before scheduling, providing data-driven support for dynamic policy generation, significantly reducing the risk of task failure or resource waste due to scheduling errors. Compared with traditional static rule-making methods, the prediction model can update the prediction results in real time based on the latest status information, thereby improving the agility and adaptability of policy adjustments.

[0055] S30, analyzing the load status of the target data source in a future time period using the analysis model to determine a task execution window, a data change rate of the target data source and satisfaction of recovery target parameters, and resource consumption and execution duration under different task types and parameter combinations;

[0056] In this embodiment, analyzing the load status of the target data source in the future time period based on the analysis model means using the prediction algorithm integrated in the analysis model to output the trend change results of the load level in the future continuous time window based on the input of the current operating data of the target data source, the execution host performance data and the network status data. The prediction dimensions of the load status usually cover indicators such as CPU occupancy, memory usage, disk I / O load, query density and write pressure. Through this prediction, a distribution map of the high and low fluctuations of the system load can be derived, and based on the preset acceptable load threshold, one or more time periods where the predicted load is lower than the threshold can be identified, thereby determining the time window suitable for task scheduling.

[0057] On this basis, the data change rate of the data source is analyzed, primarily by modeling the time series of historical data writes to predict the scale of data changes per unit time. This prediction can extract the magnitude, frequency, and periodicity of historical data changes using sliding window statistics, which serve as input for regression or time series models. The prediction results can be used to determine the level of data update activity within a specific time period, thereby assisting in determining the scheduling priority and task type for backup or synchronization tasks. Time periods with high data change rates are suitable for scheduling incremental synchronization operations, while time periods with low change rates are more suitable for performing full synchronization operations to avoid redundant data processing.

[0058] The analysis of recovery objective parameter satisfaction is based on two pre-defined dimensions: the Recovery Point Objective (RPO) and the Recovery Time Objective (RTO). It assesses whether the current task strategy can guarantee rapid system recovery in the event of a failure or anomaly. The analysis model estimates the RTO target by integrating task execution time predictions, system resource status, and network bandwidth conditions. It also assesses the probability of achieving the RPO based on the current task interval and data update frequency. If the model determines that the current task settings cannot meet the recovery objectives, it outputs corresponding adjustment suggestions for subsequent strategy generation.

[0059] Analyzing resource consumption and execution time for different task types and parameter combinations involves using an analytical model to evaluate the resource usage and execution time of various backup tasks (such as full backups, incremental backups, and log backups) under different parameter settings (such as the number of concurrent threads, compression level, and I / O priority) based on historical task data and current system status. Through simulation and deduction, resource consumption and execution time results for multiple task and parameter combinations are generated. This is used to quantify the cost and efficiency differences of different alternative solutions, providing data support for scheduling decisions.

[0060] In one implementation, a deep time series prediction model (such as Temporal Convolutional Network) is used to predict the target data source's load status for the next N hours. The model inputs include the most recent hour's operational data sequence and the execution host's performance data. The predicted value is compared with a set resource utilization threshold, and the time period with a continuous load below the threshold is selected as the task execution window.

[0061] For data change rate analysis, we extract raw data sources such as data write logs and change logs. By analyzing the data volume change curve per unit time, we use an exponentially weighted moving average (EWMA) to smooth out the noise and input it into a regression model to predict the intensity of change over the next few hours. We increase the frequency of incremental backups during periods of high change rates and schedule full backups during stable periods to reduce the number of tasks.

[0062] To assess the satisfaction of recovery target parameters, we preset an RPO of 5 minutes and an RTO of 15 minutes. The analysis model inputs include historical average task execution times, network latency, and current load status. Based on this information, we predict the completion time of new tasks. If the predicted value exceeds the RTO threshold, the current policy is marked as unsatisfactory and corresponding parameter adjustment recommendations are generated.

[0063] To analyze resource consumption and execution time for combinations of task types and parameters, a task simulator can be built based on historical training models. The simulator takes the task type, host configuration, and scheduling parameters as input and outputs a curve of estimated resource utilization and completion time. The model supports parallel simulation of multiple combinations, enabling the selection of the optimal combination under cost constraints during policy derivation.

[0064] This embodiment significantly reduces disruption to peak business hours and improves resource utilization by determining task execution windows and dynamically adjusting backup types and parameter combinations. Furthermore, analysis of RPO and RTO targets ensures recovery capabilities and mitigates potential risks. Driven by multi-objective prediction, the policy generation process is more adaptable and self-optimizing.

[0065] S40, generating a task execution policy for the target data source by comprehensively considering the task execution window, data change rate, recovery target parameter satisfaction, resource consumption, execution time, and the policy preference;

[0066] In this embodiment, a task execution strategy is generated by integrating the task execution window, data change rate, recovery target parameter satisfaction, resource consumption, execution duration, and policy preferences. This aims to build a future-oriented scheduling solution through multi-dimensional feature fusion and decision-making mechanisms. The task execution window is a continuous time period with low system load and suitable for task scheduling, as predicted by the previous analysis model. The data change rate reflects the data update activity of the target data source at different time points, which is used to determine whether incremental or full tasks are more suitable. The recovery target parameter satisfaction indicates whether the current strategy meets the system's requirements for recovery points and recovery time. If not, the strategy priority or backup frequency needs to be adjusted. Resource consumption and execution duration respectively evaluate the system resource load and task completion delay that each task type and parameter combination may generate, helping to select the optimal parameter configuration.

[0067] Policy preferences represent the biased goals set by the operations team or system, such as those favoring saving storage space, reducing backup failure rates, improving recovery speed, and limiting resource usage for core business operations. When generating task execution strategies, these preferences can be quantified as weighted parameters and applied to multi-objective optimization models. By integrating and calculating these multiple indicators, policy configuration items such as task type (e.g., full backup, incremental synchronization, log collection), task trigger time, execution host selection, resource usage limits, and compression parameters can be determined to form a complete task execution strategy.

[0068] The process of generating an execution strategy typically involves designing an objective function, setting constraints, and selecting a search strategy. The objective function might be minimizing overall resource overhead, maximizing the probability of recovery target coverage, or minimizing completion time. Constraints include task window boundaries, recovery time limits, and host resource limitations. The search strategy can employ methods such as heuristic search, reinforcement learning, and gradient optimization to discover feasible solutions that meet the preferences from the strategy space.

[0069] In one implementation, the system first filters all feasible time periods based on the task execution window output by the prediction model. Combined with the data change rate prediction results, it labels each time period with a recommended task type. Then, based on the recovery objective parameter satisfaction analysis results, it eliminates time periods and task type combinations that fail to meet RPO / RTO requirements.

[0070] Next, the task simulation module calculates the resource consumption and execution duration for each candidate task type and parameter configuration. It then constructs a multi-objective optimization function based on policy preference weights (e.g., preference for stable resources, low network utilization, or fast completion). Using methods such as greedy search or particle swarm optimization, it selects the optimal or suboptimal policy from the candidate combinations while meeting resource constraints and recovery objectives, and outputs the final task execution strategy.

[0071] During the deployment phase, the task execution strategy will be recorded and input into the task scheduling engine, including parameters such as task trigger timestamp, target execution host number, task type, resource quota, compression options, etc., to achieve intelligent scheduling and configuration of tasks.

[0072] Example: In a fintech business scenario, when generating a scheduling strategy for a payment gateway database, the system sets the lowest RPO as the primary goal based on policy preferences. A predictive model identifies the optimal task window between 1:30 and 2:00 a.m., and a simulation of task types reveals that incremental tasks within this window can be completed with minimal resource overhead. The system then automatically configures the execution host, compression ratio, and bandwidth limit, and outputs the scheduling strategy.

[0073] In the healthcare sector, given the high-frequency write cycles of electronic medical records, policy preferences are set to minimize task interference with real-time query performance. Analytical models identify nighttime as the optimal execution window and, taking into account the low data change rate, determine that executing full tasks is more appropriate. Ultimately, a task scheduling policy is generated to control the disk I / O threshold and number of concurrent threads used by tasks, enabling task configuration and deployment without impacting the primary business.

[0074] This implementation, through multidimensional data fusion and quantitative preference control, strikes a balance between resource constraints, recovery requirements, and runtime state, enabling dynamic generation and adaptive optimization of task execution strategies. Compared to traditional static scheduling mechanisms, this approach effectively avoids resource contention, reduces task failure rates, and fine-tunes strategies based on current system status and historical behavior, significantly improving scheduling efficiency and recovery capabilities while reducing manual intervention costs.

[0075] S50, dispatching the target task to the target execution host for execution according to the task execution strategy, and monitoring the execution status, resource consumption and execution duration of the target task;

[0076] In this embodiment, the target task is dispatched to the target execution host according to the task execution policy, and the execution status, resource consumption, and execution duration of the target task are monitored. This involves the entire process from policy parsing to task dispatching. The task execution policy includes key configuration items such as task type, execution time, target execution host number, resource quota limit, bandwidth limit, and task priority. The system uses the scheduling engine to parse this task execution policy and determine the task scheduling instructions based on the current cluster status map, the host's available resource pool, and scheduling priority parameters.

[0077] Target task scheduling involves injecting task resource usage parameters, generating target host scheduling signals, and initializing the task execution environment. Resource usage parameters include computing resources, memory limits, bandwidth limits, and I / O rate thresholds, and are injected into the target host execution container when scheduling instructions are issued. Task execution environment initialization can include launching the container, mounting the context, injecting scheduling parameters, and verifying data source connections.

[0078] Execution status monitoring refers to the full-cycle monitoring of tasks from initiation to completion, including real-time status identification, state change capture, failure alerts, and periodic log collection. Resource consumption monitoring includes sampling and recording metrics such as CPU usage, memory usage, network I / O, and disk writes. Execution duration measures the total time from task initialization to completion. Combined with task phase information, this can be further refined to include the duration of each phase.

[0079] To ensure data timeliness and sampling integrity, monitoring modules are typically deployed on the target execution host using lightweight probes. These modules periodically collect various operational metrics using a non-blocking mechanism and asynchronously push the results to a central logging system or analysis module. This process often combines a heartbeat mechanism with a sliding window sampling method to reduce resource consumption and enhance anomaly detection accuracy.

[0080] In one implementation, the task scheduling engine first obtains the task execution policy output by the policy generation module and parses it to identify the target execution host and the task trigger time. It then checks the target host's available resource pool to see if the CPU, memory, and bandwidth requirements specified in the policy are met. If sufficient resources are available for the current time period, the task is added to the scheduling queue and a timer trigger is set.

[0081] When the trigger time arrives, the system issues a task execution command through the container scheduler and creates a container with resource constraints on the execution host. After the initialization phase is complete, the task begins running. The scheduling engine simultaneously starts the monitoring module and activates distributed monitoring probes to record the task's current status, such as running, failed, or completed. CPU, memory, I / O, and execution time are collected at regular intervals.

[0082] During the task execution process, if there is an abnormal status or resource usage exceeds the limit, the monitoring module will immediately report it to the scheduling engine, which will determine whether to trigger the migration, interruption or rescheduling process to ensure the stable execution of the task.

[0083] Example: In healthcare, for a medical records database backup task within a hospital's diagnosis and treatment system, the system schedules the task to a low-load execution host at 2:00 AM based on a generated task execution policy. Simultaneously, parameters are implemented to limit I / O bandwidth to no more than 50 MB / s to ensure that query performance for the primary business is not affected. During the task's execution, task status and resource usage data are collected in real time. If memory usage exceeds the upper limit of 80%, the system triggers a resource downgrade mechanism to ensure successful task completion.

[0084] In the fintech sector, log backup tasks for the credit assessment engine are scheduled to suboptimal execution hosts based on predictive strategies to distribute the load. The system is configured with a policy threshold of no more than 10 minutes for execution time. During execution, the monitoring module collects the execution duration of each stage. If a stage's duration increases abnormally, the system uses the scheduling engine to adjust the task resource allocation strategy for dynamic optimization.

[0085] This embodiment combines task execution strategies with actual resource allocation and operational status monitoring mechanisms to achieve precise task scheduling and full-process monitoring. Throughout the task lifecycle, resource bottlenecks, failure risks, or execution anomalies can be detected in real time, reducing system failure rates and improving task completion rates and resource utilization. Compared to static task allocation, this mechanism implements scheduling control based on dynamic feedback, significantly enhancing the stability, agility, and scalability of the task scheduling system.

[0086] S60 , collecting the actual execution status, actual resource consumption, and actual execution duration of the target task as new historical task data, and feeding the new historical task data back to the analysis model for iterative updating.

[0087] In this embodiment, the actual execution status, actual resource consumption and actual execution time of the target task are collected as new historical task data, and the historical task data is fed back to the analysis model for iterative update, involving the whole process of parsing the operation log, archiving and reconstruction of the data structure, indicator standardization and data incremental learning. The actual execution status refers to the final execution result recorded during the execution of the task, such as success, failure, interruption or rollback, and may be accompanied by a failure reason code or exception stack information. The actual resource consumption includes multi-dimensional resource indicators such as CPU usage, memory peak, disk read and write rate, network bandwidth occupancy, etc. during the execution of the task. These indicators are usually collected in real time and structured by monitoring probes on the execution host. The actual execution time refers to the time span from the start of task scheduling to the complete exit of the task, and can be subdivided into multiple stages such as queuing, initialization, running, and completion confirmation.

[0088] This collection process utilizes a combination of asynchronous log pulling and event-driven sampling. The execution host regularly pushes the task's interim resource usage data to the central logging system. Furthermore, the system immediately captures timestamps and contextual indicators whenever the task status changes, creating event markers. After the task is completed, the central system merges the collected data based on time and event sequences to form a complete record of the actual operation. This data is then converted into standardized historical task data that conforms to the input structure of the analytical model.

[0089] Standardizing historical task data further includes field mapping, missing value handling, unit normalization, and noise removal to ensure the validity of subsequent training samples and compatibility with the model structure. Once processed, the system incrementally injects this historical task data into the analysis model and executes an update process. This update uses an incremental learning algorithm to fine-tune the parameters of the prediction, scheduling, and resource estimation modules within the analysis model, improving the model's accuracy and generalization capabilities for generating task execution strategies.

[0090] In one implementation, a lightweight resource collection module is deployed on the execution host during the target task's execution. This module collects CPU usage, memory usage, and network I / O data on a sub-second basis, and uploads this information, along with the task execution event log, to a centralized logging service. Upon task completion, the system automatically triggers a log aggregation process, merging structured logs with unstructured execution records to generate fields indicating the task's execution status, resource consumption statistics, and the duration of each stage.

[0091] The data cleaning module then normalizes these data fields, fills missing fields with time series interpolation or mean substitution, removes outliers through outlier detection, and unifies the unit format and feature encoding to form a standardized historical task data sample. After passing validity verification, this sample is injected into the training data stream, triggering the incremental training process of the analysis model and updating the weight matrix and parameter vector.

[0092] The system supports merging multiple historical samples into batches within time windows for training, improving stability and preventing overfitting. The updated model overwrites the old model and is registered with the scheduling engine, ensuring that subsequent policy generation is based on the latest data and behavioral feedback.

[0093] Example: In a healthcare scenario, after executing a regular incremental backup task for a chronic disease management database, the system detected that network bandwidth usage was high, execution duration exceeded average, and some data writes were interrupted during holidays. This execution status was recorded as a failure, and the system fed this into the model as a standardized historical task data sample. Through incremental training, the model automatically adjusted resource evaluation and scheduling strategies for subsequent holidays.

[0094] In the FinTech business, the execution records of a task that compresses and archives user transaction logs showed a significant increase in memory usage during peak trading hours. The model identified this pattern through an iterative update process and added a memory redundancy quota to the scheduling policy for similar tasks, thus avoiding the risk of future tasks failing due to insufficient memory in similar scenarios.

[0095] This embodiment constructs standardized historical task data from status data and resource metrics collected during actual task execution and feeds it back into the analysis model. This not only forms a closed loop between strategy generation and task execution, but also significantly enhances the model's responsiveness to environmental dynamics and its precision adaptability. The system can refine its estimates of resource requirements, execution time, and load pressure based on each actual execution, reducing the risk of policy deviation and improving the accuracy of policy output and overall system performance.

[0096] The present invention relates to the field of data processing technology and can be applied to business scenarios such as financial technology and medical health. The invention discloses an adaptive data processing optimization method, device, equipment and medium, including: obtaining the operating data of the target data source, the performance data of the associated execution host, network status data, historical task data, recovery target parameters and policy preferences, training a machine learning model to generate an analysis model, analyzing the load status, data change rate and satisfaction of the recovery target parameters in the future time period, and evaluating the resource consumption and execution time under different task types and parameter combinations, comprehensively analyzing the analysis results and policy preferences to generate a task execution strategy, scheduling the target task to the target execution host and monitoring its execution status, resource consumption and execution time, collecting actual execution feedback as new historical task data for iteratively updating the analysis model. The present invention constructs an analysis model by fusing multi-source system data and historical task information, combines dynamic prediction results with policy weights to generate a task execution strategy and realizes intelligent task scheduling and process monitoring, and then uses the feedback data for iterative model optimization to achieve improved task execution efficiency, rationalized resource use and enhanced model adaptability, effectively overcoming the problems of traditional static strategy rigidity and low resource utilization.

[0097] In one embodiment, the above step S10 includes:

[0098] S101, acquiring processor usage, input and output load, transaction processing rate, and data change rate from a monitoring interface of a target data source to generate the operation data;

[0099] S102, periodically collecting processor utilization, available memory capacity, storage device throughput, and network interface bandwidth occupancy through a system monitoring agent of an execution host associated with the target data source to generate the performance data;

[0100] S103, sending a probe data packet to the network link between the target data source and the candidate execution host, determining the average round-trip delay and the peak available bandwidth, and generating the network status data;

[0101] S104, extracting historical task records including task type identifier, actual execution start timestamp, actual execution end timestamp, peak resource consumption, and task execution result status code from the task management database to generate the historical task data;

[0102] S105, parsing the recovery time objective threshold and the recovery point objective threshold input by the user through the configuration interface to generate the recovery target parameters;

[0103] S106 , loading the user-predefined cost optimization weight coefficient, recovery time sensitivity coefficient, and recovery point accuracy priority parameter from the policy configuration file to generate the policy preference.

[0104] In this embodiment, in order to realize the generation of intelligent strategies for scheduling optimization and resource evaluation, it is necessary to collect data indicators with timeliness and business semantics for the target data source and its related components. The acquisition of operation data uses the system monitoring interface of the target data source as the entry point. This interface usually exposes performance indicators based on HTTP, SNMP or custom RPC protocols, including the current processor utilization rate reflecting the real-time load level of the CPU, the input and output load corresponding to the disk and cache read and write pressure, the transaction processing rate measuring the database request processing capacity per unit time, and the data change rate reflecting the incremental change trend of data in the backup task. The above data enters the central indicator collection system through polling collection or push mechanism, and is uniformly constructed into structured operation data items.

[0105] Performance data for the execution host must be collected in real time from the operating system or container orchestration system. This data is typically obtained through system monitoring agents such as Prometheus Node Exporter and Telegraf. These agent modules periodically read kernel-level metrics, including processor core utilization, available memory capacity, storage throughput, and network interface bandwidth. All metrics are cached and uploaded by time slices, allowing the scheduling system to analyze host availability and current load status.

[0106] Network status data is generated through an active probing mechanism. The system control component sends standardized probe packets to the network path between the target data source and the candidate execution host. These packets contain a unique identifier and timestamp, which are used to calculate round-trip latency (RTT) and response time. The amount of data transmitted per unit time is also measured to assess peak link bandwidth. Data packets may be based on ICMP, UDP, or TCP transport protocols. The evaluation logic uses metrics such as packet loss rate, stability, and response jitter to generate a network status dataset containing latency and bandwidth information.

[0107] Historical task data is generated through structured extraction of task execution records from the task management database. Extracted fields include task type identifiers for task classification, execution start and end timestamps for calculating task duration, peak resource consumption indicating the task's peak utilization of system resources, and task execution result status codes indicating whether the task succeeded, failed, timed out, or was aborted. All historical task records undergo field mapping and format verification before being stored to ensure they can be used as input for model training.

[0108] The recovery target parameters are entered by the user through the graphical configuration interface or the REST configuration interface. The system extracts two key thresholds, the recovery time objective (RTO) and the recovery point objective (RPO), from the configuration request. These are used to constrain the maximum recovery time and the time window for data loss, respectively, and generate a structured parameter set for subsequent model analysis and policy evaluation.

[0109] The generation of strategy preferences depends on the loading of strategy configuration files. The configuration files define the user's weights and sensitivities to different optimization objectives, including the willingness to suppress costs, the sensitivity to recovery time, and the accuracy priority of recovery points. After parsing, the system converts them into multidimensional vector form as tuning parameters in the strategy generation link.

[0110] This implementation collects multi-dimensional metrics data related to data sources, execution hosts, network channels, and historical behavior, and structures user expectations and policy preference parameters. This provides an accurate, comprehensive, and dynamic input data foundation for the subsequent construction of high-precision scheduling analysis models. Compared to fixed thresholds and static scheduling logic, this data construction approach offers stronger contextual awareness and real-time support, improving the personalized adaptability and predictive accuracy of subsequent task execution strategy generation.

[0111] In one embodiment, the above step S20 includes:

[0112] S201, dividing the processor usage and input / output load in the operation data into preset time windows to generate time-aligned operation data;

[0113] S202, matching the time-aligned running data with the actual execution start timestamp in the historical task data to generate time-matched historical task data;

[0114] S203, normalizing the storage device throughput in the performance data and the available bandwidth peak in the network status data to generate a resource supply capacity indicator with a unified dimension;

[0115] S204, extracting the correlation between the task type identifier and the peak resource consumption of the corresponding task based on the historical task data after time series matching, and constructing a task type and resource consumption mapping feature vector;

[0116] S205, performing time series decomposition on the transaction processing rate and data change rate in the time-series aligned operation data to separate the long-term trend component, the periodic seasonal component, and the random residual component;

[0117] S206, performing sliding window statistics on the average round-trip delay in the network status data and the processor utilization in the performance data to generate a joint feature matrix of dynamic load and network delay;

[0118] S207, inputting the long-term trend component and the periodic seasonal component into a long short-term memory network model, and training and generating a load window analysis sub-model;

[0119] S208, inputting the task type and resource consumption mapping feature vector and the resource supply capacity index into a gradient boosting decision tree model to train and generate a task type decision sub-model;

[0120] S209, inputting the dynamic load and network delay joint feature matrix into a random forest regression model to train and generate a resource consumption analysis sub-model;

[0121] S210: Perform cross-modal feature concatenation on the output features of the load window analysis sub-model and the input layer of the task type decision sub-model to construct an end-to-end joint training architecture;

[0122] S211, iteratively optimizing the joint training architecture using a difference-weighted loss function, and when the prediction error rate of the joint training architecture on the validation set is lower than a preset error threshold, terminating the training and saving the model parameters to generate a joint model;

[0123] S212: Encapsulate the joint model and the resource consumption analysis sub-model into an integrated analysis model that supports hot updates, deploy it to a policy generation engine, and configure a data stream interface to receive monitoring feedback data.

[0124] In this embodiment, to enable the analysis model to model the coordination of multi-source data time series and the integration of heterogeneous dimensions, it is first necessary to structurally align the processor utilization and input / output load in the operational data. The acquisition sequence is divided into time windows with a uniform time granularity, for example, by slicing continuous data segments into 5-minute intervals, generating a set of structured vectors with synchronized timestamps. This processing operation not only improves the comparability of data input but also provides a stable index for subsequent causal mapping between tasks and system states.

[0125] Matching the time-aligned runtime data with the actual execution start timestamps of each task in the historical task data is a key step in building the task behavior context. This matching logic is typically based on timestamp overlap, determining whether the runtime data time period overlaps the expected window for task initiation. This allows the extraction of task-related system state sequences and the construction of a time-matched task context data set.

[0126] To address inconsistent metrics across different data sources, it's necessary to normalize the storage device throughput in performance data and the peak available bandwidth in network status data. This normalization can be done using Z-score standardization or Min-Max scaling to convert metrics from different dimensions to a unified scale. This generates resource availability metrics that can directly contribute to model calculations, addressing the uneven impact of metrics on the computational graph.

[0127] As a key label representing task behavior patterns, task type identifiers must be correlated with peak resource consumption from historical task records and extracted. By counting the maximum resource usage of different task types during historical execution, a mapping relationship between task type and resource requirements is constructed, forming a feature vector mapping task type and resource consumption that can be used as model input. This mapping not only reflects the task load distribution but also provides a reference for estimating scheduling costs based on task classification.

[0128] To improve our understanding of data dynamics, we need to perform time series decomposition on two key metrics: transaction rate and data change rate. Decomposition methods, such as the STL (Seasonal-Trend decomposition using Loess) framework, break down time series into long-term trend components, seasonal cycle components, and residual noise components. This process provides a higher-dimensional input feature foundation for models to capture the evolving structure of the data.

[0129] Considering the coupling effect between dynamic changes in system resources and fluctuations in network performance, a sliding window approach is used to jointly calculate average round-trip latency and processor utilization, generating a joint feature matrix of dynamic load and network latency. The sliding window design can be configured to calculate statistics such as mean and standard deviation within the sliding interval to capture short-term interference patterns or network bottleneck trends.

[0130] The extracted trend and periodic components are fed into a long short-term memory (LSTM) model for training, forming a sub-model for load forecasting. The LSTM model is capable of capturing long-term dependencies and is suitable for modeling long-term sequences such as database load that evolve with business fluctuations.

[0131] In terms of resource scheduling decision modeling, the feature vectors mapping task types to resource consumption and the unified resource supply capacity index are input into the Gradient Boosting Decision Tree (GBDT) model training. This is used to construct a classification prediction path between task types and resource scheduling logic, forming a task type decision sub-model. The advantages of GBDT lie in its high modeling accuracy, sensitivity to feature importance distribution, and adaptability to scenarios with nonlinear boundaries and sparse features.

[0132] To estimate resource consumption, a random forest regression model is introduced. Taking the joint feature matrix of dynamic load and network latency as input, it learns how resource demand responds to environmental perturbations and generates a sub-model for resource consumption analysis. The ensemble structure of random forests improves prediction robustness and is suitable for handling strong inter-feature correlations and high noise interference.

[0133] Cross-modally concatenate the time series predictions output by the LSTM sub-model with the input layer of the GBDT model, integrating contextual information from workload dynamics and task classification to build an end-to-end joint training architecture. The concatenation operation must maintain consistent tensor dimensions, achieving information fusion through channel alignment or embedding alignment.

[0134] To improve generalization and error control during training, we introduced a weighted difference loss function. This function assigns weights based on task categories or sample importance, aggregates error terms, and significantly improves modeling accuracy for small or high-risk samples. During training, the prediction error rate of the validation set is continuously monitored. When the error falls below a preset threshold, training is automatically terminated, retaining the currently optimized parameters.

[0135] Finally, the model generated by the joint training architecture and the resource consumption analysis sub-model are packaged into a hot-update analysis model, capable of replacing parameters and loading structures without restarting the system. During deployment, it is mounted within the policy generation engine, and a data stream interface is configured to connect to the monitoring platform to receive real-time feedback, supporting subsequent dynamic policy adjustments and model self-updates.

[0136] This embodiment significantly enhances the modeling capabilities of complex database task load dynamics, resource consumption behavior, and scheduling window adaptability through a multi-model collaborative training mechanism and dimensional feature modeling. Cross-modal splicing training of trend prediction, task type identification, and resource estimation breaks through the bottleneck of a single model's ability to fuse multi-source heterogeneous data, enabling forward-looking assessments of the task execution environment and personalized adaptation of resource scheduling solutions. Through a differential weighted optimization strategy, the model's response accuracy to highly sensitive tasks and key parameter fluctuations is effectively controlled. Combined with a hot update mechanism, an integrated analysis architecture with real-time response and self-evolution capabilities is constructed, significantly improving the model's system adaptability and operational efficiency while ensuring prediction accuracy.

[0137] In one embodiment, the above step S30 includes:

[0138] S301, dividing the processor usage and input / output load in the currently collected operation data into preset time windows to generate real-time time-series aligned operation data;

[0139] S302, normalizing the storage device throughput in the currently collected performance data of the execution host and the available bandwidth peak in the currently collected network status data to generate a real-time resource supply capacity indicator;

[0140] S303, extracting the transaction processing rate and data change rate from the real-time time series alignment operation data, and combining them with the real-time task type identifier to construct a real-time task type and resource consumption mapping feature vector;

[0141] S304, performing time series decomposition on the transaction processing rate and data change rate in the real-time time series alignment operation data to separate a real-time long-term trend component and a real-time periodic seasonal component;

[0142] S305, inputting the real-time long-term trend component and the real-time periodic seasonal component into a joint model in the analysis model to generate a load valley probability distribution of each time window within a future preset period;

[0143] S306, screening a set of time windows that meet a low load condition based on the load valley probability distribution and a preset probability threshold;

[0144] S307: Input the real-time task type and resource consumption mapping feature vector and the real-time resource supply capability indicator into the joint model in the analysis model to generate a full backup type recommendation probability and an incremental backup type recommendation probability;

[0145] S308, obtaining the current data change rate, combining the recovery time objective threshold and the recovery point objective threshold, and determining the recovery target parameter satisfaction score;

[0146] S309, performing sliding window statistics on the average round-trip delay in the currently collected network status data and the processor utilization in the currently collected performance data to generate a joint feature matrix of real-time dynamic load and network delay;

[0147] S310: Input the real-time dynamic load and network delay joint feature matrix into the resource consumption analysis sub-model in the analysis model to generate a predicted task execution time and peak resource consumption.

[0148] In this embodiment, to achieve real-time awareness of the database's current operating status and predict scheduling opportunities, the collected operating data must first be processed to a unified time structure. By configuring fixed time window boundaries, processor utilization and input / output load are divided into consecutive time periods, generating a sequence of operating states with synchronized timestamps, and forming real-time, time-series-aligned operating data. This process ensures the stability and comparability of subsequent time series modeling.

[0149] At the resource availability awareness level, normalizing the storage device throughput from the execution host and the peak available bandwidth from network status data is key to achieving multi-dimensional metric integration. This normalization method can employ maximum-minimum normalization or logarithmic scaling to ensure that metrics of different physical dimensions have a uniform influence within the model, thereby generating a real-time resource availability indicator that reflects the upper limit of the combined service capacity of the current execution host and network channel.

[0150] The structural modeling of real-time task loads relies on two dynamic metrics: transaction processing rate and data change rate. These metrics are jointly encoded with the current task type identifier to form a feature vector that maps the real-time task type to resource consumption. This vector reflects the resource consumption behavior of different business types in the context of current data changes, providing precise parameter support for scheduling strategy selection.

[0151] The cyclical characteristics of data changes and task loads are key references for identifying scheduling windows. Using time series decomposition algorithms (such as seasonal trend decomposition or variational mode decomposition) to decompose transaction processing rates and data change rates, we can extract real-time long-term trend components and cyclical seasonal components, thereby stripping away occasional noise and short-term jitter, providing stable input for load trend forecasting.

[0152] The combined model, which incorporates these trends and periodic components into the analysis model, leverages its trained time series prediction capabilities to output a probability distribution sequence for loads within a preset future time period. These distributions are then filtered based on a preset probability threshold, retaining a set of time windows that meet low-load criteria as candidate scheduling opportunities, providing data support for avoiding system peak periods.

[0153] Backup type recommendations rely on a coordinated assessment of task attributes and system capabilities. A joint model integrates real-time task types with resource consumption mapping feature vectors and resource supply capacity indicators. The model's embedded logic generates probabilistic estimates for different task types, outputting corresponding full and incremental backup type recommendations. This enables intelligent assessment of task granularity and execution cost.

[0154] Recovery constraint satisfaction measurement is an important supplement to task scheduling adaptability assessment. Based on the current data change rate, combined with user-defined recovery time objective and recovery point objective thresholds, the deviation between the current data coverage capability and the recovery response potential is calculated to generate a recovery objective parameter satisfaction score, which is used to evaluate the risk margin of the scheduling plan.

[0155] System latency and resource congestion have a direct impact on task execution performance. By combining statistics on average round-trip latency and processor utilization through a sliding window mechanism, we construct a real-time dynamic load and network latency joint feature matrix that reflects system interaction latency and resource utilization. This matrix captures transient fluctuations in the transmission path between the execution host and the data source, providing timely support for performance risk assessment.

[0156] The above feature matrix is ​​input into the resource consumption analysis sub-model. Based on the mapping relationship between the input features and performance results acquired during the training period, the model outputs the predicted task execution duration and peak resource consumption. This supports the estimation of the performance profile of the current task plan in the actual execution environment and facilitates dynamic decision-making on task scheduling strategies.

[0157] This embodiment collaboratively processes operational data, performance data, and network status data in a unified time window, and integrates multi-dimensional inputs such as task type, data change rate, recovery target, and delayed load characteristics. The analysis model can predict future load trends, task adaptation windows, and execution cost parameters based on real-time status perception. Based on the linked reasoning of the joint model and the resource consumption sub-model, it can dynamically generate matching execution time, backup type, and resource configuration recommendations for tasks, and quantitatively measure their recovery security, thereby achieving systematic prediction of the execution timing, type, and performance impact of database tasks. This mechanism significantly improves the real-time adaptability and resource allocation accuracy of task scheduling, reduces the probability of conflicts and the risk of recovery failure during execution, and effectively improves the overall backup efficiency and recovery reliability of the database.

[0158] In one embodiment, the above step S40 includes:

[0159] S401, based on the cost optimization weight, recovery time sensitivity coefficient, and resource utilization threshold in the policy preference, a multi-dimensional weighted processing is performed on the load status indicator, data change rate, recovery target parameter satisfaction, resource consumption, and execution time of the task execution window to generate a policy priority vector;

[0160] S402, constructing a multi-dimensional policy constraint set by using the task execution window as a time constraint, the recovery target parameter satisfaction as a task type constraint, and the resource consumption and performance parameters of the execution host as resource constraints;

[0161] S403, using a multi-objective optimization module to solve the policy priority vector and the multi-dimensional policy constraint set to generate a candidate policy set that meets preset cost, time, and resource constraints;

[0162] S404: For each policy in the candidate policy set, verify the conflict of its task execution window, the compatibility of the backup type and storage medium, and the matching degree of resource consumption and network bandwidth, eliminate policies that do not meet the conditions, and generate a feasible policy set;

[0163] S405, selecting an optimal strategy from the set of feasible strategies as the task execution strategy according to the priority strategy in the strategy preference;

[0164] S406: Convert the task execution policy into an executable instruction set including execution time, backup type, target host and resource configuration parameters.

[0165] In this embodiment, in order to derive specific executable scheduling decisions from multi-dimensional input factors, it is first necessary to express variables such as the task execution window, data change rate, recovery target parameter satisfaction, resource consumption, and execution time in a unified vector, and perform priority mapping in combination with the weight information in the policy preference. The cost optimization weight contained in the policy preference reflects the degree of emphasis on resource conservation, the recovery time sensitivity coefficient reflects the requirement for timely response to tasks, and the resource utilization threshold defines the upper limit of what the system can bear. After multiplying and weighting the above factors with the task status indicators, a policy priority vector is formed, which serves as a target reference for optimization decisions.

[0166] The task execution window defines the time range within which operations can be executed. The satisfaction of recovery target parameters indirectly limits the execution mode or backup type that can be used for the task. Resource consumption and execution host performance parameters define the optional execution entities and concurrency capacity. Based on this, three types of policy constraints are constructed: time, type, and resource. These are combined to form a multi-dimensional policy constraint set, representing the constraints on the scheduling solution.

[0167] During the solution phase, a multi-objective optimization module is introduced to jointly solve the policy priority vector and the constraint set. This module uses a weighted objective function and a heuristic search algorithm or evolutionary computation method, such as multi-objective search based on particle swarm optimization or greedy screening based on the Pareto frontier, to generate a set of candidate policies that simultaneously meet resource, time, and policy preferences.

[0168] The candidate policy set requires further feasibility verification. This process sequentially determines whether the selected task execution window overlaps with existing scheduled tasks, checks whether the recommended backup type is compatible with the current storage media capacity and access method, and evaluates whether the task's estimated resource consumption is within the network bandwidth capacity. Through logical condition screening, policies with potential conflicts or violations of system configuration constraints are eliminated, resulting in a final set of feasible policies.

[0169] When selecting the optimal strategy from the set of feasible strategies, the priority mechanism is used. For example, if the priority preference is shortest recovery time, the strategy with the shortest predicted execution time and meeting the RTO target is selected. If the priority preference is lowest resource cost, the strategy with the lowest peak resource consumption and lowest network utilization is prioritized. The final selected strategy becomes the basis for scheduling the current task.

[0170] The selected task execution policy is converted into dispatchable control information, which must be encapsulated into an instruction set that includes the scheduling time, task type identifier, target host IP address, and resource allocation parameters (such as the maximum number of CPU cores, memory limit, and estimated bandwidth). This instruction set is called through the interface between the policy engine and the task scheduling system, driving the task to be executed by the appropriate resource node at the optimal time.

[0171] This embodiment jointly models multi-dimensional task status indicators and policy preference parameters, and constructs a composite constraint structure of time, resources, and task types, which can efficiently generate and filter scheduling strategies while preserving user intent. By combining optimization solving with compatibility verification, it can be ensured that the selected strategy not only meets the trade-off goals of cost, performance, and recovery requirements, but also has system-level feasibility. The results are further converted into a structured instruction set, which enhances the enforceability of the strategy in the task scheduling system. This mechanism significantly improves the intelligence level and response efficiency of database task scheduling, avoids problems such as resource contention, scheduling conflicts, and recovery failures, and reduces the complexity of manual intervention and policy adjustment while improving the level of automation.

[0172] In one embodiment, the above step S50 includes:

[0173] S501, parsing the executable instruction set in the task execution policy, and extracting the task triggering timestamp, backup type identifier, target host address and resource configuration parameters;

[0174] S502, sending a resource verification request to the target execution host corresponding to the target host address to verify whether the current available memory capacity, processor utilization and network bandwidth of the target execution host meet the requirements of the resource configuration parameters;

[0175] S503, if the resource configuration parameter requirements are not met, a dynamic rescheduling process is triggered to reselect a backup execution host that meets the resource configuration parameter requirements as the target execution host;

[0176] S504, transmitting the control script and data slices of the target task to the target execution host according to the task trigger timestamp;

[0177] S505, when the task trigger timestamp is reached, starting the control script through the remote execution interface and recording the task start timestamp;

[0178] S506, periodically collecting task process status data of the target execution host;

[0179] S507, determining the deviation between the actual resource consumption of the target execution host in executing the target task and the predicted resource consumption, and generating a resource consumption anomaly alarm;

[0180] S508, recording the task execution progress percentage, the amount of data processed, and the remaining estimated duration, and generating a task execution log with a timestamp;

[0181] S509: When it is detected that the task execution time exceeds the preset tolerance range of the predicted value, an execution timeout alarm is triggered and root cause analysis data is recorded.

[0182] In this embodiment, before scheduling a task, the executable instruction set contained in the task execution policy is parsed to extract key control parameters for controlling task scheduling. These parameters include the task trigger timestamp, which indicates the specific execution time; the backup type identifier, which specifies the functional intent of the task, such as full or incremental backup; the target host address, which is used to target the task execution node; and resource configuration parameters, such as allocated memory capacity, number of processor threads, or maximum bandwidth usage, which are used to set the resource range required for task execution.

[0183] To ensure stable execution of the task on the target host, the system sends a resource verification request to the target host address, collecting operational status indicators such as available memory capacity, processor utilization, and network bandwidth. The system then compares these real-time indicators with the resource configuration parameters required by the task execution strategy to determine whether the target host can meet the resource requirements of the current task.

[0184] If the target execution host experiences insufficient memory, a fully loaded CPU, or network bandwidth congestion, the system automatically initiates a rescheduling process, searching for a backup execution host that meets the resource configuration parameters and replacing it with the new target execution host. This process relies on the real-time monitoring and coordination capabilities of the scheduling system, enabling rapid target node switching without disrupting the scheduling plan.

[0185] Once the target execution host that meets the resource requirements is determined, the system will transmit the task control script and the required data fragments through a secure channel to the preset path of the target execution host before the task trigger timestamp, and prepare the remote interface call instructions.

[0186] When the system time reaches the task trigger timestamp, the scheduling system executes the control script through a remote interface call to start the target task. At startup, the system writes the task start timestamp into the scheduling log for subsequent status analysis and backtracking.

[0187] During task execution, the system periodically polls the target execution host for task process status data, including process survival status, resource usage changes, and file write progress, and compares this data with the predicted status model. By comparing the task's actual resource consumption with the predicted resource consumption, the deviation between the two is calculated. If the deviation exceeds a threshold, a resource consumption anomaly alert is generated, allowing the management system to promptly warn of potential resource bottlenecks or task failure risks.

[0188] The system also records dynamic indicators such as the progress percentage of task execution, the amount of data processed, and the remaining estimated time, and adds an accurate timestamp to each record to generate a structured task execution log to support subsequent audits and execution performance analysis.

[0189] If the system detects that the cumulative execution time of a task has exceeded the upper limit of the predicted execution time and the tolerance threshold, it will immediately generate an execution timeout alarm and record information such as resources, network, and host status related to the execution anomaly as root cause analysis data, providing a basis for subsequent operation and maintenance troubleshooting and model optimization.

[0190] This embodiment effectively ensures the adaptability of task scheduling and the compatibility of system resources by converting the executable instruction set in the task execution policy into structured scheduling instructions, combined with real-time resource verification and automatic rescheduling mechanisms. Periodic process monitoring and resource anomaly alert mechanisms further enhance state observability and risk warning capabilities during task execution. Structured logging and deviation analysis during execution not only improve operational transparency but also provide a real-world data foundation for subsequent model iterations.

[0191] In one embodiment, the above step S60 includes:

[0192] S601: Fill missing values ​​and remove outliers for the actual execution time, actual resource consumption, and task completion status code in the task execution log to generate a standardized historical task record.

[0193] S602: Time-series alignment is performed on the standardized historical task records with the task type identifier, resource configuration parameters, and dynamic load and network delay joint feature matrix in the task execution strategy to construct complete historical task data including a feature vector mapping task type and resource consumption, actual resource consumption, and execution result status code.

[0194] S603, extracting task type identification, task execution duration, peak resource consumption, and network delay characteristics from the complete historical task data according to a preset sample screening strategy to generate an incremental training sample set;

[0195] S604: Input the incremental training sample set into the joint model and the resource consumption analysis sub-model in the analysis model, and update the weight parameters of the joint model and the resource consumption analysis sub-model through the incremental learning module;

[0196] S605: Encapsulate the updated joint model and resource consumption analysis sub-model into an online updateable analysis model.

[0197] In this embodiment, after task execution is complete, key operational indicators are extracted from the task execution log to continuously optimize the subsequent scheduling and modeling processes. These indicators include actual execution time, actual resource consumption, and the task completion status code. The actual execution time is typically derived from the difference between the start and end timestamps recorded by the task scheduling system. Actual resource consumption can be extracted from the resource usage records of the task process during execution by the monitoring system. The task completion status code is uniformly transmitted back by the control script at the termination stage, indicating whether the task was completed normally or interrupted by an error.

[0198] After collection, this data requires preprocessing to ensure data consistency and validity in the subsequent modeling process. First, missing values ​​in the data are interpolated or mean-filled, and outliers that significantly deviate from the expected range are statistically removed to form a standardized historical task record with uniform structure and stable values. Missing values ​​can be filled using a sliding window averaging strategy or historical median substitution. Outlier removal can use IQR box plots or triple standard deviation strategies to identify outliers.

[0199] After standardization, the historical task records must be time-aligned with the static configuration parameters (such as the task type identifier and resource configuration parameters) recorded in the original task execution policy, as well as the dynamic load and network delay joint feature matrix generated during scheduling, to ensure the consistency of the input data for model learning. By linking timestamp matching with the unique task identifier, a complete historical task data structure with a one-to-one correspondence can be created, including the feature vector mapping task type and resource consumption, actual resource consumption, actual execution duration, and final execution status.

[0200] To prevent the model from introducing excessive redundant or low-quality samples during incremental training, a sample screening strategy must be designed to perform fine-grained screening based on task execution status, resource fluctuation range, and training set sparsity. This strategy may include retaining only task records with normal status codes, prioritizing samples with significant resource fluctuations, and scaling up the sampling of tasks with large execution duration variations. The resulting incremental training sample set will include structured input fields such as task type identification, task execution duration, peak resource consumption, and network latency characteristics.

[0201] This incremental training sample set is fed into the joint model and the resource consumption analysis sub-model, respectively, and the internal model weight parameters are updated via an online incremental learning mechanism. The incremental learning module must minimize the error for new input samples without resetting the original model structure and weights, while preserving the original model's ability to fit the historical dataset. This can be accomplished using an elastic net regularized update mechanism based on buffered samples, or using an approximate second-order optimizer such as AdamW for online weight optimization.

[0202] After completing the model weight update, the updated joint model and resource consumption analysis sub-model are packaged into an integrated model version that supports hot updates and re-registered with the policy generation engine. This packaging operation must preserve model version control, inference interface stability, and transactional consistency during model switching to ensure smooth replacement of the old and new models and rollback.

[0203] Example: In corporate credit operations, core databases contain high-value information such as daily credit approval data, corporate performance records, and risk ratings, requiring regular, high-reliability backups. To this end, the system incorporates an intelligent scheduling mechanism to automatically generate and optimize backup task execution strategies. First, the system uses the database's performance monitoring module to collect information about processor utilization, disk read / write load, concurrent transaction rates, and changes in the number of records during peak business periods to build operational data. Resource proxy nodes collect information about the corresponding execution hosts, including current CPU usage, available memory, storage channel throughput, and network traffic, to generate performance data. Furthermore, the system uses network probing to obtain link latency and peak available bandwidth between the source database node and the candidate host, supplementing network status data. Historical task data is extracted from task scheduling logs and backup result records, covering task type, run time, resource consumption, and execution status. Combined with the business continuity level defined by the user in the policy configuration interface (e.g., a high level requires RTO < 10 minutes, RPO < 1 hour) and cost priorities, recovery target parameters and policy preferences are generated.

[0204] The system then executes the model building process. First, all historical resource data is segmented by time window, timestamps aligned, and mapped to task results. Processor and network capabilities are normalized to generate resource supply indicators, and task types are modeled by combining type and resource consumption mapping. After the transaction rate and data change rate are decomposed into time series, long-term trends and cyclical patterns are extracted and then input into the long short-term memory network (LSTM) to form a load prediction sub-model. Other models such as gradient boosting trees and random forest models are used for task type recommendation and resource consumption analysis, respectively. Finally, they are spliced ​​together through a joint training architecture to optimize the loss function and generate a complete integrated analysis model.

[0205] In actual operation, the system checks current status data every morning and inputs it into the model. If the model predicts that the database load is lowest between 2:00 and 2:30 AM, combined with a low data change rate and a high recovery objective satisfaction score, and the backup task execution cost is low based on the preference weight, then this period is included in the policy candidate list. The execution policy engine then generates a set of candidate policies, combining current resource availability with the probability of task type recommendation. Each policy is then verified for time window conflicts, storage compatibility, and resource demand exceeding limits. Ultimately, the optimal incremental backup policy is selected and distributed via an executable instruction set to the backup host with the most abundant resources.

[0206] Before a task is triggered, the system initiates a resource check on the target host. If a resource bottleneck, such as insufficient memory or network congestion, is detected, a backup host is automatically selected as a replacement. The task scheduling engine transmits the control script and shard data and initiates the backup execution via remote command at the predicted time, recording the timestamp. During execution, the system periodically monitors the progress percentage, resource deviation, and remaining estimated duration. If the task execution duration significantly exceeds the model's prediction, the system automatically logs an alert and saves an analysis report of the influencing factors.

[0207] After a task is completed, the system collects the task log, cleans it of outliers and missing fields, and extracts elements such as the actual task type, execution results, resource consumption, and network status to form a standardized historical record. This data is used to incrementally supplement training samples and enter the model update process, optimizing model parameters through incremental learning. This ensures that the model is more accurate and adaptable in the subsequent generation of strategies for similar credit tasks.

[0208] In the digital medical record system of a tertiary general hospital, core data such as medical images, test results, and hospitalization records change dynamically every day. The scheduling timing and execution path of backup tasks require refined management to improve disaster recovery capabilities.

[0209] The system extracts the current computing resource pressure, transaction processing rate, and data update frequency from the operation monitoring of the EMR data source to form a high-dimensional operation data set. The corresponding execution host collects performance status through the hospital IT monitoring platform, obtains storage IO bottlenecks, memory idle ratio, network transmission bandwidth, and generates performance data; the physical link test between the backup server obtains network status information, covering average latency and burst packet loss rate. Historical task execution records are exported from the task scheduling log maintained by the Medical Information Department, including resource consumption trends and success rates of daily full or incremental snapshot tasks. Combined with the hospital's strong recovery requirements for electrocardiogram, CT imaging and other data with an RTO of less than 5 minutes and an RPO of no more than 15 minutes, a policy preference matrix is ​​generated.

[0210] During the model training phase, operational data is aligned with historical execution time points and normalized. Load fluctuation patterns are analyzed through time series decomposition to identify long-term and short-term dynamic trends. These trends are then fed into an LSTM model to build a load forecasting module. Task types and peak resource consumption are modeled and fed into a GBDT to generate recommended task type capabilities. Network status and resource capability vectors are fed into a RF model to generate time and resource consumption estimates. After multimodal concatenation and unified training and optimization, multiple models are packaged into an analytical model that supports online updates.

[0211] One morning, the hospital's task scheduling engine analyzes and predicts that the load is lowest between 1:30 AM and 2:00 AM, CT data changes are minimal, and disk capacity and bandwidth meet the task configuration requirements. This period is marked as the preferred task window. Based on the task type recommendation probability, incremental backup is preferred. The system further evaluates the likelihood that the task will meet RTO / RPO requirements and generates a policy priority vector. The policy optimization module comprehensively matches resources, timeliness, and cost to ultimately select the optimal task execution strategy.

[0212] After the policy is delivered to the target backup host as an executable instruction, if a pre-task resource check reveals that the server's CPU usage has exceeded the threshold, the system automatically switches to another partitioned storage host and executes the data snapshot task via a remote script call mechanism. During execution, the system continuously records progress status, resource deviation trends, and latency analysis parameters. If the predicted and actual deviations exceed a safe tolerance, an alert is issued and the suspected task configuration is recorded in the log.

[0213] Upon completion, the system collects task status codes, completion time, and resource usage, integrates execution strategies and network status, and constructs high-quality training samples to feed into the incremental learning pipeline, iteratively optimizing the analytical model. After a period of continuous optimization, the model can accurately determine the backup priority of each data type, adjust task type strategies in real time, and support the coordination and prediction of concurrent tasks across multiple departments and data sources.

[0214] This embodiment significantly improves the representativeness and effectiveness of training samples by collecting real-world operational data during task execution and performing standardization, alignment, and screening. It also optimizes the analytical model in real time through online incremental training, enabling model parameters to continuously adapt to current environmental changes, thereby enhancing the model's generalization and decision-making accuracy in complex task scheduling scenarios. The model's hot update packaging mechanism ensures the security and business continuity of replacing old models with new ones, overall improving the real-time nature of resource forecasting, the intelligence of task scheduling, and the system's adaptability to variable loads.

[0215] In one embodiment, an adaptive data processing optimization device is provided, which corresponds one-to-one to the adaptive data processing optimization method in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of the adaptive data processing and optimization device of the present invention. It includes a data acquisition module 10, a model training module 20, a prediction and analysis module 30, a strategy generation module 40, a task scheduling module 50, and a model update module 60. Each functional module is described in detail as follows:

[0216] The data acquisition module 10 is used to obtain the operating data of the target data source, the performance data of the execution host associated with the target data source, the network status data, the historical task data, the recovery target parameters and the policy preferences;

[0217] A model training module 20 is configured to construct and train a machine learning model based on the operation data, the performance data of the execution host, the network status data, and the historical task data to generate an analysis model;

[0218] The prediction analysis module 30 is used to analyze the load status of the target data source in a future time period using the analysis model to determine the task execution window, the data change rate of the target data source and the satisfaction of the recovery target parameters, as well as the resource consumption and execution time under different task types and parameter combinations;

[0219] A strategy generation module 40 is configured to generate a task execution strategy for the target data source by comprehensively considering the task execution window, data change rate, recovery target parameter satisfaction, resource consumption, execution time, and strategy preference;

[0220] The task scheduling module 50 is used to schedule the target task to the target execution host for execution according to the task execution strategy, and monitor the execution status, resource consumption and execution duration of the target task;

[0221] The model updating module 60 is used to collect the actual execution status, actual resource consumption and actual execution duration of the target task as new historical task data, and feed the new historical task data back to the analysis model for iterative updating.

[0222] In one embodiment, the data acquisition module 10 is specifically configured to:

[0223] Obtaining processor usage, input and output load, transaction processing rate, and data change rate from a monitoring interface of a target data source to generate the operation data;

[0224] Periodically collecting processor utilization, available memory capacity, storage device throughput, and network interface bandwidth occupancy through a system monitoring agent of an execution host associated with the target data source to generate the performance data;

[0225] Sending a probe data packet to a network link between the target data source and the candidate execution host, determining an average round-trip delay and a peak available bandwidth, and generating the network status data;

[0226] Extracting historical task records including task type identification, actual execution start timestamp, actual execution end timestamp, peak resource consumption and task execution result status code from a task management database to generate the historical task data;

[0227] Parsing the recovery time objective threshold and the recovery point objective threshold input by the user through the configuration interface to generate the recovery objective parameters;

[0228] The cost optimization weight coefficient, recovery time sensitivity coefficient, and recovery point accuracy priority parameter predefined by the user are loaded from the policy configuration file to generate the policy preference.

[0229] In one embodiment, the model training module 20 is specifically configured to:

[0230] Segmenting the processor usage and input / output load in the operation data according to a preset time window to generate time-aligned operation data;

[0231] Matching the time-aligned running data with the actual execution start timestamp in the historical task data to generate time-matched historical task data;

[0232] Normalizing the storage device throughput in the performance data and the available bandwidth peak in the network status data to generate a resource supply capacity indicator with a unified dimension;

[0233] Based on the historical task data after time series matching, extract the correlation between the task type identifier and the peak resource consumption of the corresponding task, and construct a task type and resource consumption mapping feature vector;

[0234] Performing time series decomposition on the transaction processing rate and data change rate in the time-series aligned operating data to separate a long-term trend component, a periodic seasonal component, and a random residual component;

[0235] Perform sliding window statistics on the average round-trip delay in the network status data and the processor utilization in the performance data to generate a joint feature matrix of dynamic load and network delay;

[0236] Inputting the long-term trend component and the periodic seasonal component into a long short-term memory network model to train and generate a load window analysis sub-model;

[0237] Inputting the task type and resource consumption mapping feature vector and resource supply capacity index into the gradient boosting decision tree model to train and generate a task type decision sub-model;

[0238] Inputting the dynamic load and network delay joint feature matrix into a random forest regression model to train and generate a resource consumption analysis sub-model;

[0239] Perform cross-modal feature concatenation on the output features of the load window analysis sub-model and the input layer of the task type decision sub-model to construct an end-to-end joint training architecture;

[0240] Iteratively optimizing the joint training architecture using a difference-weighted loss function, terminating the training and saving the model parameters to generate a joint model when the prediction error rate of the joint training architecture on the validation set is lower than a preset error threshold;

[0241] The joint model and the resource consumption analysis sub-model are encapsulated into an integrated analysis model that supports hot updates, deployed to a policy generation engine, and a data stream interface is configured to receive monitoring feedback data.

[0242] In one embodiment, the prediction analysis module 30 is specifically configured to:

[0243] The processor usage and input and output load in the currently collected operation data are divided into preset time windows to generate real-time time-series aligned operation data;

[0244] Normalize the storage device throughput in the currently collected performance data of the execution host and the available bandwidth peak in the currently collected network status data to generate a real-time resource supply capacity indicator;

[0245] Extracting the transaction processing rate and data change rate from the real-time time series alignment operation data, and combining them with the real-time task type identifier to construct a real-time task type and resource consumption mapping feature vector;

[0246] Performing time series decomposition on the transaction processing rate and the data change rate in the real-time time series alignment operation data to separate a real-time long-term trend component and a real-time periodic seasonal component;

[0247] Inputting the real-time long-term trend component and the real-time periodic seasonal component into a joint model in the analysis model to generate a load valley probability distribution for each time window within a future preset period;

[0248] Filtering a set of time windows that meet low load conditions based on the load valley probability distribution and a preset probability threshold;

[0249] Input the real-time task type and resource consumption mapping feature vector and the real-time resource supply capability index into the joint model in the analysis model to generate a full backup type recommendation probability and an incremental backup type recommendation probability;

[0250] Obtain the current data change rate and, based on the recovery time objective threshold and recovery point objective threshold, determine the recovery objective parameter satisfaction score;

[0251] Perform sliding window statistics on the average round-trip delay in the currently collected network status data and the processor utilization in the currently collected performance data to generate a joint feature matrix of real-time dynamic load and network delay;

[0252] The real-time dynamic load and network delay joint feature matrix is ​​input into the resource consumption analysis sub-model in the analysis model to generate the predicted task execution time and peak resource consumption.

[0253] In one embodiment, the policy generation module 40 is specifically configured to:

[0254] Based on the cost optimization weight, recovery time sensitivity coefficient and resource utilization threshold in the policy preference, the load status indicator, data change rate, recovery target parameter satisfaction, resource consumption and execution time of the task execution window are multi-dimensionally weighted to generate a policy priority vector;

[0255] The task execution window is used as a time constraint, the satisfaction of the recovery target parameters is used as a task type constraint, and the resource consumption and performance parameters of the execution host are used as resource constraints to construct a multi-dimensional policy constraint set;

[0256] A multi-objective optimization module is used to solve the policy priority vector and the multi-dimensional policy constraint set to generate a candidate policy set that meets the preset cost, time and resource constraints;

[0257] For each policy in the candidate policy set, verify the conflict of its task execution window, the compatibility of the backup type and storage medium, and the matching degree of resource consumption and network bandwidth. Eliminate the policies that do not meet the conditions and generate a feasible policy set.

[0258] Selecting an optimal strategy from the set of feasible strategies as the task execution strategy according to the priority strategy in the strategy preference;

[0259] The task execution strategy is converted into an executable instruction set including execution time, backup type, target host and resource configuration parameters.

[0260] In one embodiment, the task scheduling module 50 is specifically configured to:

[0261] Parsing the executable instruction set in the task execution strategy and extracting the task trigger timestamp, backup type identifier, target host address and resource configuration parameters;

[0262] Sending a resource verification request to the target execution host corresponding to the target host address to verify whether the current available memory capacity, processor utilization and network bandwidth of the target execution host meet the requirements of the resource configuration parameters;

[0263] If the resource configuration parameter requirements are not met, a dynamic rescheduling process is triggered to reselect a backup execution host that meets the resource configuration parameter requirements as the target execution host;

[0264] According to the task trigger timestamp, the control script and data slices of the target task are transmitted to the target execution host;

[0265] When the task trigger timestamp is reached, the control script is started through the remote execution interface, and the task start timestamp is recorded;

[0266] Periodically collecting task process status data of the target execution host;

[0267] Determine the deviation between the actual resource consumption of the target execution host in executing the target task and the predicted resource consumption, and generate a resource consumption anomaly alarm;

[0268] Record the task execution progress percentage, amount of data processed, and remaining estimated duration, and generate a task execution log with a timestamp;

[0269] When it is detected that the task execution time exceeds the preset tolerance range of the predicted value, an execution timeout alarm is triggered and the root cause analysis data is recorded.

[0270] In one embodiment, the model updating module 60 is specifically configured to:

[0271] Fill missing values ​​and remove outliers in the actual execution time, actual resource consumption, and task completion status code in the task execution log to generate standardized historical task records;

[0272] Performing time-series alignment on the standardized historical task records with the task type identifier, resource configuration parameters, and dynamic load and network delay joint feature matrix in the task execution strategy to construct complete historical task data including the task type and resource consumption mapping feature vector, actual resource consumption, and execution result status code;

[0273] According to a preset sample screening strategy, task type identification, task execution duration, peak resource consumption and network delay characteristics are extracted from the complete historical task data to generate an incremental training sample set;

[0274] Inputting the incremental training sample set into the joint model and the resource consumption analysis sub-model in the analysis model, and updating the weight parameters of the joint model and the resource consumption analysis sub-model through the incremental learning module;

[0275] The updated joint model and resource consumption analysis sub-model are encapsulated into an analysis model that can be updated online.

[0276] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, memory, network interface and database connected via a system bus. The processor of the computer device is used to provide determination and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the server side of an adaptive data processing optimization method.

[0277] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide determination and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the user side of an adaptive data processing optimization method.

[0278] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0279] Obtaining operating data of a target data source, performance data of an execution host associated with the target data source, network status data, historical task data, recovery target parameters, and policy preferences;

[0280] Building and training a machine learning model based on the operation data, the performance data of the execution host, the network status data, and the historical task data to generate an analysis model;

[0281] By using the analysis model, the load status of the target data source in a future time period is analyzed to determine the task execution window, the data change rate of the target data source and the satisfaction of the recovery target parameters, as well as the resource consumption and execution time under different task types and parameter combinations;

[0282] Generate a task execution policy for the target data source based on the task execution window, data change rate, recovery target parameter satisfaction, resource consumption, execution time, and policy preference;

[0283] Dispatching the target task to the target execution host for execution according to the task execution strategy, and monitoring the execution status, resource consumption and execution duration of the target task;

[0284] The actual execution status, actual resource consumption, and actual execution duration of the target task are collected as new historical task data, and the new historical task data are fed back to the analysis model for iterative updating.

[0285] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0286] Obtaining operating data of a target data source, performance data of an execution host associated with the target data source, network status data, historical task data, recovery target parameters, and policy preferences;

[0287] Building and training a machine learning model based on the operation data, the performance data of the execution host, the network status data, and the historical task data to generate an analysis model;

[0288] By using the analysis model, the load status of the target data source in a future time period is analyzed to determine the task execution window, the data change rate of the target data source and the satisfaction of the recovery target parameters, as well as the resource consumption and execution time under different task types and parameter combinations;

[0289] Generate a task execution policy for the target data source based on the task execution window, data change rate, recovery target parameter satisfaction, resource consumption, execution time, and policy preference;

[0290] Dispatching the target task to the target execution host for execution according to the task execution strategy, and monitoring the execution status, resource consumption and execution duration of the target task;

[0291] The actual execution status, actual resource consumption, and actual execution duration of the target task are collected as new historical task data, and the new historical task data are fed back to the analysis model for iterative updating.

[0292] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0293] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0294] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0295] It should be noted that if any software tools or components other than those of the Company appear in the embodiments of this application, they are merely for illustration and do not represent actual use. The above embodiments are intended only to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some of the technical features therein with equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. An adaptive data processing optimization method, characterized in that: The following steps are involved: Obtaining operating data of a target data source, performance data of an execution host associated with the target data source, network status data, historical task data, recovery target parameters, and policy preferences; Building and training a machine learning model based on the operation data, the performance data of the execution host, the network status data, and the historical task data to generate an analysis model; By using the analysis model, the load status of the target data source in a future time period is analyzed to determine the task execution window, the data change rate of the target data source and the satisfaction of the recovery target parameters, as well as the resource consumption and execution time under different task types and parameter combinations; Generate a task execution policy for the target data source based on the task execution window, data change rate, recovery target parameter satisfaction, resource consumption, execution time, and policy preference; Dispatching the target task to the target execution host for execution according to the task execution strategy, and monitoring the execution status, resource consumption and execution duration of the target task; The actual execution status, actual resource consumption, and actual execution duration of the target task are collected as new historical task data, and the new historical task data are fed back to the analysis model for iterative updating.

2. The adaptive data processing optimization method according to claim 1, wherein: Obtaining the target data source's operating data, performance data of the execution host associated with the target data source, network status data, historical task data, recovery target parameters, and policy preferences, including: Obtaining processor usage, input and output load, transaction processing rate, and data change rate from a monitoring interface of a target data source to generate the operation data; Periodically collecting processor utilization, available memory capacity, storage device throughput, and network interface bandwidth occupancy through a system monitoring agent of an execution host associated with the target data source to generate the performance data; Sending a probe data packet to a network link between the target data source and the candidate execution host, determining an average round-trip delay and a peak available bandwidth, and generating the network status data; Extracting historical task records including task type identification, actual execution start timestamp, actual execution end timestamp, peak resource consumption and task execution result status code from a task management database to generate the historical task data; Parsing the recovery time objective threshold and the recovery point objective threshold input by the user through the configuration interface to generate the recovery objective parameters; The cost optimization weight coefficient, recovery time sensitivity coefficient, and recovery point accuracy priority parameter predefined by the user are loaded from the policy configuration file to generate the policy preference.

3. The adaptive data processing optimization method according to claim 1, wherein: Building and training a machine learning model based on the operation data, the performance data of the execution host, the network status data, and the historical task data to generate an analysis model, including: Segmenting the processor usage and input / output load in the operation data according to a preset time window to generate time-aligned operation data; Matching the time-aligned running data with the actual execution start timestamp in the historical task data to generate time-matched historical task data; Normalizing the storage device throughput in the performance data and the available bandwidth peak in the network status data to generate a resource supply capacity indicator with a unified dimension; Based on the historical task data after time series matching, extract the correlation between the task type identifier and the peak resource consumption of the corresponding task, and construct a task type and resource consumption mapping feature vector; Performing time series decomposition on the transaction processing rate and data change rate in the time-series aligned operating data to separate a long-term trend component, a periodic seasonal component, and a random residual component; Perform sliding window statistics on the average round-trip delay in the network status data and the processor utilization in the performance data to generate a joint feature matrix of dynamic load and network delay; Inputting the long-term trend component and the periodic seasonal component into a long short-term memory network model to train and generate a load window analysis sub-model; Inputting the task type and resource consumption mapping feature vector and resource supply capacity index into the gradient boosting decision tree model to train and generate a task type decision sub-model; Inputting the dynamic load and network delay joint feature matrix into a random forest regression model to train and generate a resource consumption analysis sub-model; Perform cross-modal feature concatenation on the output features of the load window analysis sub-model and the input layer of the task type decision sub-model to construct an end-to-end joint training architecture; Iteratively optimizing the joint training architecture using a difference-weighted loss function, terminating the training and saving the model parameters to generate a joint model when the prediction error rate of the joint training architecture on the validation set is lower than a preset error threshold; The joint model and the resource consumption analysis sub-model are encapsulated into an integrated analysis model that supports hot updates, deployed to a policy generation engine, and a data stream interface is configured to receive monitoring feedback data.

4. The adaptive data processing optimization method according to claim 1, wherein: By using the analysis model, the load status of the target data source in the future time period is analyzed to determine the task execution window, the data change rate of the target data source and the satisfaction of the recovery target parameters, as well as the resource consumption and execution time under different task types and parameter combinations, including: The processor usage and input and output load in the currently collected operation data are divided into preset time windows to generate real-time time-series aligned operation data; Normalize the storage device throughput in the currently collected performance data of the execution host and the available bandwidth peak in the currently collected network status data to generate a real-time resource supply capacity indicator; Extracting the transaction processing rate and data change rate from the real-time time series alignment operation data, and combining them with the real-time task type identifier to construct a real-time task type and resource consumption mapping feature vector; Performing time series decomposition on the transaction processing rate and the data change rate in the real-time time series alignment operation data to separate a real-time long-term trend component and a real-time periodic seasonal component; Inputting the real-time long-term trend component and the real-time periodic seasonal component into a joint model in the analysis model to generate a load valley probability distribution for each time window within a future preset period; Filtering a set of time windows that meet low load conditions based on the load valley probability distribution and a preset probability threshold; Input the real-time task type and resource consumption mapping feature vector and the real-time resource supply capability index into the joint model in the analysis model to generate a full backup type recommendation probability and an incremental backup type recommendation probability; Obtain the current data change rate and, based on the recovery time objective threshold and recovery point objective threshold, determine the recovery objective parameter satisfaction score; Perform sliding window statistics on the average round-trip delay in the currently collected network status data and the processor utilization in the currently collected performance data to generate a joint feature matrix of real-time dynamic load and network delay; The real-time dynamic load and network delay joint feature matrix is ​​input into the resource consumption analysis sub-model in the analysis model to generate the predicted task execution time and peak resource consumption.

5. The adaptive data processing optimization method according to claim 1, wherein: A task execution strategy is generated for the target data source based on the task execution window, data change rate, recovery target parameter satisfaction, resource consumption, execution time, and strategy preference, including: Based on the cost optimization weight, recovery time sensitivity coefficient and resource utilization threshold in the policy preference, the load status indicator, data change rate, recovery target parameter satisfaction, resource consumption and execution time of the task execution window are multi-dimensionally weighted to generate a policy priority vector; The task execution window is used as a time constraint, the satisfaction of the recovery target parameters is used as a task type constraint, and the resource consumption and performance parameters of the execution host are used as resource constraints to construct a multi-dimensional policy constraint set; A multi-objective optimization module is used to solve the policy priority vector and the multi-dimensional policy constraint set to generate a candidate policy set that meets the preset cost, time and resource constraints; For each policy in the candidate policy set, verify the conflict of its task execution window, the compatibility of the backup type and storage medium, and the matching degree of resource consumption and network bandwidth. Eliminate the policies that do not meet the conditions and generate a feasible policy set. Selecting an optimal strategy from the set of feasible strategies as the task execution strategy according to the priority strategy in the strategy preference; The task execution strategy is converted into an executable instruction set including execution time, backup type, target host and resource configuration parameters.

6. The adaptive data processing optimization method according to claim 1, wherein: Schedule the target task to the target execution host for execution according to the task execution strategy, and monitor the execution status, resource consumption, and execution duration of the target task, including: Parsing the executable instruction set in the task execution strategy and extracting the task trigger timestamp, backup type identifier, target host address and resource configuration parameters; Sending a resource verification request to the target execution host corresponding to the target host address to verify whether the current available memory capacity, processor utilization and network bandwidth of the target execution host meet the requirements of the resource configuration parameters; If the resource configuration parameter requirements are not met, a dynamic rescheduling process is triggered to reselect a backup execution host that meets the resource configuration parameter requirements as the target execution host; According to the task trigger timestamp, the control script and data slices of the target task are transmitted to the target execution host; When the task trigger timestamp is reached, the control script is started through the remote execution interface, and the task start timestamp is recorded; Periodically collecting task process status data of the target execution host; Determine the deviation between the actual resource consumption of the target execution host in executing the target task and the predicted resource consumption, and generate a resource consumption anomaly alarm; Record the task execution progress percentage, amount of data processed, and remaining estimated duration, and generate a task execution log with a timestamp; When it is detected that the task execution time exceeds the preset tolerance range of the predicted value, an execution timeout alarm is triggered and the root cause analysis data is recorded.

7. The adaptive data processing optimization method according to claim 1, wherein: Collecting the actual execution status, actual resource consumption, and actual execution duration of the target task as new historical task data, and feeding the new historical task data back to the analysis model for iterative updating, including: Fill missing values ​​and remove outliers in the actual execution time, actual resource consumption, and task completion status code in the task execution log to generate standardized historical task records; Performing time-series alignment on the standardized historical task records with the task type identifier, resource configuration parameters, and dynamic load and network delay joint feature matrix in the task execution strategy to construct complete historical task data including the task type and resource consumption mapping feature vector, actual resource consumption, and execution result status code; According to a preset sample screening strategy, task type identification, task execution duration, peak resource consumption and network delay characteristics are extracted from the complete historical task data to generate an incremental training sample set; Inputting the incremental training sample set into the joint model and the resource consumption analysis sub-model in the analysis model, and updating the weight parameters of the joint model and the resource consumption analysis sub-model through the incremental learning module; The updated joint model and resource consumption analysis sub-model are encapsulated into an analysis model that can be updated online.

8. An adaptive data processing optimization device, characterized in that: The adaptive data processing optimization device comprises: A data acquisition module is used to obtain the operating data of the target data source, the performance data of the execution host associated with the target data source, network status data, historical task data, recovery target parameters and policy preferences; A model training module is used to build and train a machine learning model based on the operation data, the performance data of the execution host, the network status data and the historical task data to generate an analysis model; A prediction analysis module is used to analyze the load status of the target data source in a future time period through the analysis model to determine the task execution window, the data change rate of the target data source and the satisfaction of the recovery target parameters, as well as the resource consumption and execution time under different task types and parameter combinations; A strategy generation module, configured to generate a task execution strategy for the target data source by comprehensively considering the task execution window, data change rate, recovery target parameter satisfaction, resource consumption, execution time, and strategy preference; The task scheduling module is used to schedule the target task to the target execution host for execution according to the task execution strategy, and monitor the execution status, resource consumption and execution time of the target task; The model updating module is used to collect the actual execution status, actual resource consumption and actual execution duration of the target task as new historical task data, and feed the new historical task data back to the analysis model for iterative updating.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and an adaptive data processing optimization program stored in the memory and capable of running on the processor. When the adaptive data processing optimization program is executed by the processor, the steps of the adaptive data processing optimization method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The storage medium stores an adaptive data processing optimization program, which, when executed by a processor, implements the steps of the adaptive data processing optimization method according to any one of claims 1 to 7.

Images