AI-based automatic optimization method and system for big data distributed computing tasks

Abstract

The present invention provides an AI-based automatic optimization method and system for big data distributed computing tasks, which relates to the field of AI technology, including constructing a multimodal spatiotemporal feature perception network to extract the spatiotemporal features of tasks and resources, and training a hierarchical hybrid decision network to perform global task scheduling and local resource allocation. The global policy network adopts a hierarchical deep reinforcement learning architecture, combined with Monte Carlo tree search and priority experience replay to generate decisions; the local execution network optimizes local deployment based on graph neural networks and multi-agent collaborative learning. In addition, a distributed anomaly detection network is deployed to monitor performance in real time, and adaptive tuning is achieved through enhanced transfer learning, and finally the perception network is updated through knowledge distillation to achieve model evolution. The present invention can effectively improve the execution efficiency and resource utilization of big data distributed computing tasks, and reduce system operating costs.

Description

Technical Field

[0001] The present invention relates to AI technology, and in particular to an AI-based automatic optimization method and system for big data distributed computing tasks. Background Art

[0002] With the advent of the big data era, distributed computing has become a key technology for processing massive data. In order to improve the efficiency and resource utilization of distributed computing systems, task scheduling and resource allocation are crucial. Traditional task scheduling methods usually rely on manual experience or rule-based strategies, which are difficult to adapt to complex and changing computing environments and dynamic task loads. In addition, traditional resource allocation schemes often lack flexibility and cannot be dynamically adjusted according to task requirements and resource availability, resulting in resource waste or task delays.

[0003] Difficulty in effectively handling complex task dependencies and data locality: In big data distributed computing scenarios, there are usually complex dependencies between tasks, and data is usually distributed on different computing nodes. Traditional task scheduling methods are difficult to effectively handle these dependencies and data locality, resulting in low task execution efficiency.

[0004] Lack of adaptability to dynamic environments: The computing environment and task loads usually change dynamically. Traditional task scheduling and resource allocation solutions are difficult to adapt to these changes, resulting in decreased system performance.

[0005] Insufficient exception handling capabilities: In distributed computing systems, hardware failures, network congestion and other abnormal situations often occur. Traditional task scheduling and resource allocation solutions lack effective exception handling mechanisms, making it difficult to ensure system stability and reliability. Summary of the invention

[0006] The embodiments of the present invention provide an AI-based automatic optimization method and system for big data distributed computing tasks, which can solve the problems in the prior art.

[0007] According to a first aspect of the embodiments of the present invention,

[0008] Provides an AI-based automatic optimization method for big data distributed computing tasks, including:

[0009] Construct a multimodal spatiotemporal feature perception network, wherein the multimodal spatiotemporal feature perception network uses hierarchical bidirectional long short-term memory units to construct a temporal feature extraction layer, and converts the task arrival time, task completion time limit, data dependency, and computing load intensity in the temporal feature extraction layer into a multidimensional temporal feature sequence; a graph attention network is used to construct a spatial feature extraction layer, and the geographical distribution, network topology, computing power, storage capacity, and bandwidth resources of the computing nodes in the spatial feature extraction layer are converted into dynamic graph features; the multidimensional temporal feature sequence and the dynamic graph features are adaptively fused through a dynamic weighted self-attention mechanism to generate a heterogeneous feature tensor with spatiotemporal correlation, and the expression ability of the heterogeneous feature tensor is enhanced through a contrastive learning method;

[0010] Based on the heterogeneous feature tensor training, a hierarchical hybrid decision network is constructed, wherein the hierarchical hybrid decision network includes a global policy network and a local execution network, wherein the global policy network adopts a hierarchical deep reinforcement learning architecture, extracts hierarchical features of task queue status, resource utilization status, and system load status through a transform encoder, and generates a global decision for task scheduling and resource allocation in combination with a Monte Carlo tree search and a priority experience replay mechanism; the local execution network adopts a graph neural network structure, optimizes the task deployment location and resource allocation ratio in a local computing node cluster based on the global decision, and realizes dynamic coordination of local decisions through a multi-agent collaborative learning mechanism;

[0011] The hierarchical hybrid decision network is used to perform adaptive scheduling, and a distributed anomaly detection network is deployed during task execution. The distributed anomaly detection network builds a normal mode baseline of multi-dimensional performance indicators based on a variational autoencoder, monitors performance fluctuations and load balancing status in real time through an anomaly detector, and predicts resource demand trends in combination with a graph time series neural network. When an anomaly is detected, an adaptive tuning mechanism is triggered. The adaptive tuning mechanism uses a reinforcement transfer learning method to generate task migration strategies and resource reallocation plans, and achieves system optimization through progressive task migration and elastic resource adjustment. The optimization results are used to update the multimodal spatiotemporal feature perception network through knowledge distillation to achieve knowledge accumulation and model evolution.

[0012] Based on the heterogeneous feature tensor training, a hierarchical hybrid decision network is constructed, wherein the hierarchical hybrid decision network includes a global policy network and a local execution network, wherein the global policy network adopts a hierarchical deep reinforcement learning architecture, extracts hierarchical features of task queue status, resource utilization status, and system load status through a transform encoder, and combines Monte Carlo tree search and priority experience playback mechanism to generate global decisions for task scheduling and resource allocation, including:

[0013] Constructing a hierarchical hybrid decision network, wherein the hierarchical hybrid decision network includes a global policy network and a local execution network, wherein the global policy network adopts a hierarchical deep reinforcement learning architecture; inputting the heterogeneous feature tensor into a transform encoder, and extracting hierarchical features through a multi-head self-attention mechanism of the transform encoder, wherein the hierarchical features are used to describe the multi-level association among the task queue state, the resource utilization state, and the system load state;

[0014] Inputting the hierarchical features into the global policy network, generating global decisions for task scheduling and resource allocation based on the hierarchical features; constructing a Monte Carlo tree search, using the global decision as an initial strategy to guide the search process, and obtaining a candidate decision sequence through tree search expansion; recording a state evaluation value at a tree search node, and selecting an optimal decision branch based on the state evaluation value;

[0015] Design a priority experience playback mechanism to record decision trajectories and system feedback information; calculate the importance weight of the decision trajectory based on the temporal difference error and construct a priority sampling probability distribution; sample high-value training data from the experience pool according to the priority sampling probability distribution, and use the high-value training data to optimize the global strategy network;

[0016] The decision output of the global policy network is sent to the local execution network, and the local execution network generates specific scheduling instructions and resource allocation solutions.

[0017] The local execution network adopts a graph neural network structure, optimizes the task deployment location and resource allocation ratio in the local computing node cluster based on the global decision, and realizes dynamic coordination of local decisions through a multi-agent collaborative learning mechanism, including:

[0018] Receiving a resource allocation plan of the global decision network, wherein the resource allocation plan includes a target area for task deployment, a reference ratio for resource allocation, and constraints for load balancing;

[0019] Under the guidance of the resource allocation scheme, a graph neural network structure of a local computing node cluster is constructed, wherein the nodes in the graph neural network structure include computing resource capabilities, current load status, and task execution records, and the connection edges between nodes include network bandwidth, transmission delay, and connection stability;

[0020] Based on the graph neural network structure, a graph neural network is used to extract the state features of the local computing node cluster, and the node information is aggregated through the graph convolution layer and the attention layer to generate a feature vector reflecting the node computing capacity and load distribution;

[0021] According to the feature vector, a matching score between the task to be deployed and the computing node is calculated, wherein the matching score comprehensively considers the remaining resources of the node and the quality of the network connection to generate a priority ranking of the task deployment;

[0022] Based on the priority sorting, the task deployment location is optimized, and the resource allocation ratio of each task on the target computing node is determined through iterative adjustment, where the resource allocation ratio includes CPU ratio, memory ratio, and bandwidth ratio;

[0023] The task deployment location and the resource allocation ratio are input into a multi-agent collaborative learning system. Each local execution network in the multi-agent collaborative learning system acts as an independent agent and makes decisions based on local observation information and the status of neighboring nodes. Dynamic coordination of local decisions is achieved through the multi-agent collaborative learning system.

[0024] The hierarchical hybrid decision network is used to perform adaptive scheduling, and a distributed anomaly detection network is deployed during task execution. The distributed anomaly detection network builds a normal mode baseline of multi-dimensional performance indicators based on a variational autoencoder, monitors performance fluctuations and load balancing status in real time through an anomaly detector, and predicts resource demand trends in combination with a graph time series neural network, including:

[0025] The hierarchical hybrid decision network is used to perform adaptive scheduling, and the hierarchical hybrid decision network formulates an initial task deployment plan and a resource allocation strategy according to the real-time status information of the computing nodes as a benchmark reference for subsequent anomaly detection;

[0026] Based on the initial task deployment plan, a distributed anomaly detection network is deployed during the task execution process. The distributed anomaly detection network collects and standardizes the performance indicator data of each node in real time, including processor usage, memory occupancy, network throughput, and task response time, and generates a performance indicator vector for anomaly detection;

[0027] Input the performance indicator vector into a variational autoencoder, and construct a normal mode baseline of the multi-dimensional performance indicator through the variational autoencoder, wherein the encoder maps the performance indicator vector to a latent variable space to generate probability distribution parameters, and the decoder reconstructs the sampled latent variables back to the original feature space to establish a benchmark model for the normal operation of the system;

[0028] Based on the benchmark model, the anomaly detector continuously monitors performance fluctuations and load balancing status, calculates the deviation of real-time performance indicators from the normal mode baseline, and combines the load distribution status evaluation to generate a state description vector containing the degree of anomaly and load information; the state description vector is used as input and combined with the graph time series neural network to predict resource demand trends.

[0029] When an anomaly is detected, the adaptive tuning mechanism is triggered. The adaptive tuning mechanism uses the reinforcement transfer learning method to generate task migration strategies and resource reallocation plans, and realizes system optimization through progressive task migration and elastic resource adjustment. The optimization results are used to update the multimodal spatiotemporal feature perception network through knowledge distillation to realize knowledge accumulation and model evolution, including:

[0030] A multimodal spatiotemporal feature perception network is used to monitor the status of computing nodes in real time and generate system anomaly detection results; when the multimodal spatiotemporal feature perception network detects an anomaly, the anomaly detection result is input into an adaptive tuning mechanism, and the adaptive tuning mechanism constructs a state space and an action space, wherein the state space includes anomaly detection results and real-time performance indicators, and the action space includes optional operations of task migration and resource allocation;

[0031] The adaptive tuning mechanism constructs a tuning network based on the reinforcement transfer learning method. The tuning network includes a main strategy network and a target strategy network. The state space information is input into the main strategy network to generate an action probability distribution. The action value is evaluated through the target strategy network, and the strategy knowledge in the historical optimization experience is transferred to the current scene. The task migration strategy and resource reallocation plan are output;

[0032] According to the task migration strategy, the task migration is performed in a progressive manner, the resource status of the target node is evaluated and a migration checkpoint is established, the incremental data transmission is performed according to the planned migration sequence, the running status of the task after the migration is verified and the system topology information is updated;

[0033] According to the resource reallocation plan, perform elastic resource adjustment, dynamically adjust the number of processor cores, memory capacity limit and network bandwidth allocation of the migrated task, and generate optimized resource configuration parameters;

[0034] Feedback the optimization results of task migration and resource adjustment through knowledge distillation, build a teacher network to save optimization experience, use the student network to learn the latest strategy, guide knowledge migration by calculating feature similarity loss function, and integrate historical experience and new knowledge;

[0035] The fused knowledge is updated to the multimodal spatiotemporal feature perception network to optimize the feature extraction and recognition capabilities of the anomaly detection model, improve the accuracy of subsequent anomaly detection, and form a closed-loop evolutionary mechanism of detection-tuning-optimization.

[0036] Feedback the optimization results of task migration and resource adjustment through knowledge distillation, build a teacher network to save optimization experience, use the student network to learn the latest strategy, guide knowledge migration by calculating the feature similarity loss function, and integrate historical experience and new knowledge, including:

[0037] Collecting task migration data and resource adjustment data in the computing system, extracting performance differences before and after migration, resource consumption changes, and task completion time from the task migration data, extracting resource allocation plans, performance improvement degrees, and system load status from the resource adjustment data, and characterizing the extracted data to generate an optimization result vector;

[0038] Constructing a teacher network for storing optimization experience, wherein the feature mapping layer of the teacher network receives the optimization result vector, uses a multi-layer perceptron to extract feature associations, highlights key information through an attention mechanism, and uses residual connections to retain original features. The experience encoding layer of the teacher network uses a recurrent unit and a memory module to store the processed optimization experience;

[0039] The teacher network organizes the stored optimization experience into a mapping structure of scenario-strategy-effect, establishes a decision tree index optimization path, uses incremental learning to update the optimization experience, eliminates inefficient experience based on a value evaluation mechanism, maintains the correlation between experiences, and forms an optimization experience knowledge base;

[0040] Construct a student network for learning new scenario strategies. The student network uses a lightweight convolutional layer to extract real-time input features, selects key features through a pooling layer and an attention mechanism, and the strategy generation module of the student network generates candidate solutions based on the extracted features and uses an evaluation network to predict the execution effect.

[0041] Calculate the similarity between the optimized experience features output by the teacher network and the strategy features generated by the student network to obtain feature similarity; construct a knowledge distillation loss function based on the feature similarity, including soft label cross entropy loss and strategy consistency constraint, and adjust the degree of knowledge transfer through a temperature parameter;

[0042] Using course learning to transfer knowledge, selecting training samples based on the output of the knowledge distillation loss function, constructing a learning sequence with increasing difficulty, establishing a knowledge graph to guide the migration path, and selectively inheriting optimization experience after evaluating knowledge relevance;

[0043] The new optimization strategy learned by the student network is fed back to the optimization experience knowledge base, and the stored optimization experience is updated through the experience encoding layer of the teacher network to achieve dynamic fusion of historical experience and new scene knowledge.

[0044] According to a second aspect of the embodiments of the present invention,

[0045] Provides an AI-based automatic optimization system for big data distributed computing tasks, including:

[0046] The first unit is used to construct a multimodal spatiotemporal feature perception network, wherein the multimodal spatiotemporal feature perception network uses a hierarchical bidirectional long short-term memory unit to construct a temporal feature extraction layer, and converts the task arrival time, task completion time limit, data dependency, and computing load intensity in the temporal feature extraction layer into a multidimensional temporal feature sequence; a graph attention network is used to construct a spatial feature extraction layer, and the geographical distribution, network topology, computing power, storage capacity, and bandwidth resources of the computing nodes in the spatial feature extraction layer are converted into dynamic graph features; the multidimensional temporal feature sequence and the dynamic graph features are adaptively fused through a dynamic weight self-attention mechanism to generate a heterogeneous feature tensor with spatiotemporal correlation, and the expression ability of the heterogeneous feature tensor is enhanced through a contrastive learning method;

[0047] The second unit is used to train a hierarchical hybrid decision network based on the heterogeneous feature tensor, wherein the hierarchical hybrid decision network includes a global policy network and a local execution network, wherein the global policy network adopts a hierarchical deep reinforcement learning architecture, extracts hierarchical features of task queue status, resource utilization status and system load status through a transform encoder, and generates a global decision for task scheduling and resource allocation in combination with a Monte Carlo tree search and a priority experience replay mechanism; the local execution network adopts a graph neural network structure, optimizes the task deployment location and resource allocation ratio in a local computing node cluster based on the global decision, and realizes dynamic coordination of local decisions through a multi-agent collaborative learning mechanism;

[0048] The third unit is used to perform adaptive scheduling using the hierarchical hybrid decision network, deploy a distributed anomaly detection network during task execution, the distributed anomaly detection network builds a normal mode baseline of multi-dimensional performance indicators based on a variational autoencoder, monitors performance fluctuations and load balancing status in real time through an anomaly detector, and predicts resource demand trends in combination with a graph time series neural network. When an anomaly is detected, an adaptive tuning mechanism is triggered, and the adaptive tuning mechanism uses a reinforcement transfer learning method to generate task migration strategies and resource reallocation plans, and achieves system optimization through progressive task migration and elastic resource adjustment. The optimization results are used to update the multimodal spatiotemporal feature perception network through knowledge distillation to achieve knowledge accumulation and model evolution.

[0049] According to a third aspect of the embodiments of the present invention,

[0050] An electronic device is provided, comprising:

[0051] processor;

[0052] a memory for storing processor-executable instructions;

[0053] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0054] A fourth aspect of the embodiments of the present invention is:

[0055] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0056] The beneficial effects of this application are as follows:

[0057] 1. Improve resource utilization efficiency and task execution efficiency: The present invention can more accurately perceive task characteristics, resource status and system load through a multimodal spatiotemporal feature perception network and a hierarchical hybrid decision network, thereby achieving better task scheduling and resource allocation. The global strategy network combines Monte Carlo tree search and priority experience playback mechanism to generate more global and long-term decisions; the local execution network dynamically coordinates local decisions through a multi-agent collaborative learning mechanism to further improve resource utilization efficiency and task execution efficiency.

[0058] 2. Enhance system stability and robustness: The distributed anomaly detection network deployed by the present invention can monitor system performance in real time, and predict resource demand trends in combination with graph time series neural networks, thereby discovering potential problems in advance. When an anomaly is detected, the adaptive tuning mechanism can generate effective task migration strategies and resource reallocation plans through enhanced transfer learning methods, realize progressive task migration and elastic resource adjustment, ensure stable operation of the system, and improve its robustness in dealing with emergencies.

[0059] 3. Achieve continuous learning and optimization: The present invention uses the knowledge distillation method to feed back the optimization results to the multimodal spatiotemporal feature perception network to achieve knowledge accumulation and model evolution. This continuous learning mechanism enables the system to continuously adapt to new tasks and environmental changes, and continuously improve its performance over time. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a flow chart of an automatic optimization method for big data distributed computing tasks based on AI according to an embodiment of the present invention;

[0061] Figure 2 This is a schematic diagram of the first AI-based automatic optimization technology flow for big data distributed computing tasks according to an embodiment of the present invention;

[0062] Figure 3 This is a schematic diagram of the second AI-based automatic optimization technology flow for big data distributed computing tasks according to an embodiment of the present invention;

[0063] Figure 4 This is a schematic diagram of the structure of an automatic optimization system for big data distributed computing tasks based on AI according to an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0066] Figure 1 FIG. 1 is a flow chart of an automatic optimization method for big data distributed computing tasks based on AI according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0067] S11. Construct a multimodal spatiotemporal feature perception network, wherein the multimodal spatiotemporal feature perception network uses hierarchical bidirectional long short-term memory units to construct a temporal feature extraction layer, and converts the task arrival time, task completion time limit, data dependency, and computing load intensity in the temporal feature extraction layer into a multidimensional temporal feature sequence; uses a graph attention network to construct a spatial feature extraction layer, and converts the geographical distribution, network topology, computing power, storage capacity, and bandwidth resources of the computing nodes in the spatial feature extraction layer into dynamic graph features; adaptively fuses the multidimensional temporal feature sequence and the dynamic graph features through a dynamic weight self-attention mechanism to generate a heterogeneous feature tensor with spatiotemporal correlation, and enhances the expression ability of the heterogeneous feature tensor through a contrastive learning method;

[0068] S12. Based on the heterogeneous feature tensor training, a hierarchical hybrid decision network is constructed, wherein the hierarchical hybrid decision network includes a global policy network and a local execution network, wherein the global policy network adopts a hierarchical deep reinforcement learning architecture, extracts hierarchical features of task queue status, resource utilization status, and system load status through a transform encoder, and generates a global decision for task scheduling and resource allocation in combination with a Monte Carlo tree search and a priority experience replay mechanism; the local execution network adopts a graph neural network structure, optimizes the task deployment location and resource allocation ratio within a local computing node cluster based on the global decision, and realizes dynamic coordination of local decisions through a multi-agent collaborative learning mechanism;

[0069] S13. Utilize the hierarchical hybrid decision network to perform adaptive scheduling, and deploy a distributed anomaly detection network during task execution. The distributed anomaly detection network builds a normal mode baseline of multi-dimensional performance indicators based on a variational autoencoder, monitors performance fluctuations and load balancing status in real time through anomaly detectors, and predicts resource demand trends in combination with graph time series neural networks. When an anomaly is detected, an adaptive tuning mechanism is triggered. The adaptive tuning mechanism uses a reinforcement transfer learning method to generate task migration strategies and resource reallocation plans, and achieves system optimization through progressive task migration and elastic resource adjustment. The optimization results are used to update the multimodal spatiotemporal feature perception network through knowledge distillation to achieve knowledge accumulation and model evolution.

[0070] In an optional embodiment, a hierarchical hybrid decision network is trained based on the heterogeneous feature tensor, and the hierarchical hybrid decision network includes a global policy network and a local execution network, wherein the global policy network adopts a hierarchical deep reinforcement learning architecture, extracts hierarchical features of task queue status, resource utilization status, and system load status through a transform encoder, and combines Monte Carlo tree search and priority experience playback mechanism to generate global decisions for task scheduling and resource allocation, including:

[0071] Constructing a hierarchical hybrid decision network, wherein the hierarchical hybrid decision network includes a global policy network and a local execution network, wherein the global policy network adopts a hierarchical deep reinforcement learning architecture; inputting the heterogeneous feature tensor into a transform encoder, and extracting hierarchical features through a multi-head self-attention mechanism of the transform encoder, wherein the hierarchical features are used to describe the multi-level association among the task queue state, the resource utilization state, and the system load state;

[0072] Inputting the hierarchical features into the global policy network, generating global decisions for task scheduling and resource allocation based on the hierarchical features; constructing a Monte Carlo tree search, using the global decision as an initial strategy to guide the search process, and obtaining a candidate decision sequence through tree search expansion; recording a state evaluation value at a tree search node, and selecting an optimal decision branch based on the state evaluation value;

[0073] Design a priority experience playback mechanism to record decision trajectories and system feedback information; calculate the importance weight of the decision trajectory based on the temporal difference error and construct a priority sampling probability distribution; sample high-value training data from the experience pool according to the priority sampling probability distribution, and use the high-value training data to optimize the global strategy network;

[0074] The decision output of the global policy network is sent to the local execution network, and the local execution network generates specific scheduling instructions and resource allocation solutions.

[0075] A method for training hierarchical hybrid decision networks based on heterogeneous feature tensors for optimizing task scheduling and resource allocation is proposed. The method includes a global policy network and a local execution network.

[0076] First, a hierarchical hybrid decision network is constructed. The network consists of a global policy network and a local execution network. The global policy network uses a hierarchical deep reinforcement learning architecture to generate global decisions for task scheduling and resource allocation. The local execution network is responsible for converting global decisions into specific scheduling instructions and resource allocation plans.

[0077] Then, the input heterogeneous feature tensor is processed. The heterogeneous feature tensor contains information such as task queue status, resource utilization status, and system load status. This information is usually represented in different formats and dimensions. For example, the task queue can be represented by task ID, priority, arrival time, etc., the resource utilization status can be represented by CPU usage, memory occupancy, etc., and the system load status can be represented by average response time, request throughput, etc. These heterogeneous features are processed using a transform encoder. The transform encoder uses a multi-head self-attention mechanism to extract hierarchical features. The multi-head self-attention mechanism allows multiple attention heads to focus on different parts of the input features in parallel, so that the multi-level associations between features can be captured. For example, one attention head can focus on the priority of tasks in the task queue, and another attention head can focus on the availability of resources, thereby associating task priority with resource availability. The extracted hierarchical features can more comprehensively describe the relationship between the task queue status, resource utilization status, and system load status.

[0078] Next, the hierarchical features are input into the global policy network. The global policy network generates global decisions on task scheduling and resource allocation based on these features. For example, the global policy network can decide to assign high-priority tasks to idle resources or to delay the execution of low-priority tasks.

[0079] In order to further optimize the global decision, a Monte Carlo tree search is constructed. The decision generated by the global policy network is used as the initial policy to guide the search process. The candidate decision sequence is obtained by tree search expansion. The state evaluation value is recorded at each node of the tree search. The state evaluation value is used to evaluate the pros and cons of the current state. For example, the state evaluation value can be calculated using a pre-trained evaluation model or heuristic rules. The optimal decision branch is selected based on the state evaluation value. For example, the node with the highest state evaluation value is selected as the next expansion node.

[0080] In order to train the global policy network, a priority experience replay mechanism is designed. Record decision trajectories and system feedback information. The decision trajectory includes a series of decisions and state transitions, and the system feedback information includes rewards or penalties. Calculate the importance weight of the decision trajectory based on the temporal difference error. The temporal difference error measures the impact of the current decision on future rewards. Construct a priority sampling probability distribution. The priority sampling probability distribution is constructed based on the importance weight of the decision trajectory. The higher the importance weight, the higher the probability of being sampled. For example, a decision trajectory that causes a system crash will have a high importance weight, so it will be sampled first. Sample high-value training data from the experience pool according to the priority sampling probability distribution. These high-value training data are used to optimize the global policy network.

[0081] Finally, the decision output of the global policy network is sent to the local execution network. The local execution network generates specific scheduling instructions and resource allocation plans. For example, if the global policy network decides to assign task A to resource B, the local execution network generates specific instructions to transfer the data of task A to resource B and start the execution of task A on resource B.

[0082] For example, suppose there is a system with three tasks and two resources. The task queue status includes the priority and computation time of each task, the resource utilization status includes the availability and processing power of each resource, and the system load status includes the current average response time. The transform encoder extracts hierarchical features, such as the association between task priority and resource availability. The global policy network generates global decisions based on these features, such as assigning high-priority, short-computation-time tasks to idle, high-capacity resources. Monte Carlo tree search further optimizes global decisions, such as selecting the best one by simulating different scheduling schemes. The priority experience replay mechanism records decision trajectories and system feedback information, such as recording the completion time and resource utilization of each task. Based on this information, the global policy network is trained to better adapt to different system load conditions. The local execution network converts global decisions into specific scheduling instructions and resource allocation schemes, such as assigning tasks to specified resources and setting the execution parameters of tasks.

[0083] The solution of this application can:

[0084] Improve resource utilization: Through the collaborative work of the global policy network and the local execution network, resources can be allocated more effectively, resource waste can be avoided, and resource utilization can be improved. Reduce system latency: Through the priority experience replay mechanism and Monte Carlo tree search, the task scheduling strategy can be optimized, the task waiting time can be reduced, and the system latency can be reduced. Enhance system stability: Through hierarchical feature extraction and layered deep reinforcement learning architecture, changes in system status can be better captured and corresponding adjustments can be made to enhance system stability.

[0085] In an optional implementation, the local execution network adopts a graph neural network structure, optimizes the task deployment location and resource allocation ratio in the local computing node cluster based on the global decision, and realizes dynamic coordination of local decisions through a multi-agent collaborative learning mechanism, including:

[0086] Receiving a resource allocation plan of the global decision network, wherein the resource allocation plan includes a target area for task deployment, a reference ratio for resource allocation, and constraints for load balancing;

[0087] Under the guidance of the resource allocation scheme, a graph neural network structure of a local computing node cluster is constructed, wherein the nodes in the graph neural network structure include computing resource capabilities, current load status, and task execution records, and the connection edges between nodes include network bandwidth, transmission delay, and connection stability;

[0088] Based on the graph neural network structure, a graph neural network is used to extract the state features of the local computing node cluster, and the node information is aggregated through the graph convolution layer and the attention layer to generate a feature vector reflecting the node computing capacity and load distribution;

[0089] According to the feature vector, a matching score between the task to be deployed and the computing node is calculated, wherein the matching score comprehensively considers the remaining resources of the node and the quality of the network connection to generate a priority ranking of the task deployment;

[0090] Based on the priority sorting, the task deployment location is optimized, and the resource allocation ratio of each task on the target computing node is determined through iterative adjustment, where the resource allocation ratio includes CPU ratio, memory ratio, and bandwidth ratio;

[0091] The task deployment location and the resource allocation ratio are input into a multi-agent collaborative learning system. Each local execution network in the multi-agent collaborative learning system acts as an independent agent and makes decisions based on local observation information and the status of neighboring nodes. Dynamic coordination of local decisions is achieved through the multi-agent collaborative learning system.

[0092] The local execution network optimizes task deployment and resource allocation based on global decisions to achieve efficient cluster resource management. This method uses graph neural networks and multi-agent collaborative learning to dynamically adjust task deployment and resource allocation within the local computing node cluster to adapt to changing load and resource conditions.

[0093] First, the local execution network receives the resource allocation plan from the global decision network. The plan includes the target area for task deployment, the benchmark ratio of resource allocation, and the load balancing constraints. For example, the global decision network specifies that task A is deployed in area 1, and the benchmark ratio of resource allocation is 50% for CPU, 30% for memory, and 20% for bandwidth, and requires that the load balancing degree of area 1 be maintained above 90%.

[0094] Next, the local execution network builds a graph neural network structure of the local computing node cluster based on the received resource allocation plan. The nodes in this structure represent computing nodes, which contain information such as computing resource capabilities, current load status, and task execution records. The connection edges between nodes represent the connection relationship between nodes, which contain information such as network bandwidth, transmission delay, and connection stability. For example, the computing resource capabilities of node 1 are 16 CPU cores, 64GB memory size, 60% CPU load, and 40% memory load. The connection bandwidth between node 1 and node 2 is 10Gbps, the transmission delay is 2ms, and the connection stability is 99%.

[0095] Then, the local execution network uses the graph neural network to extract the state features of the local computing node cluster based on the constructed graph neural network structure. The node information is aggregated through the graph convolution layer and the attention layer to generate a feature vector that reflects the node computing power and load distribution. For example, the feature vector of node 1 contains information such as its CPU resources, memory resources, current load status, and the quality of its connection with its neighboring nodes.

[0096] Subsequently, the local execution network calculates the matching score between the task to be deployed and the computing node based on the generated feature vector. This score takes into account the remaining resources of the node and the quality of the network connection to generate a priority ranking for task deployment. For example, if the matching score of task A with node 1 is 0.8 and the matching score with node 2 is 0.6, then task A will be deployed on node 1 first.

[0097] Next, the local execution network optimizes the task deployment location based on the generated priority ranking. The resource allocation ratio of each task on the target computing node is determined through iterative adjustment, including CPU ratio, memory ratio, and bandwidth ratio. For example, task A is finally deployed on node 1, and the resource ratio allocated is 60% for CPU, 20% for memory, and 20% for bandwidth.

[0098] Finally, the local execution network inputs the determined task deployment location and resource allocation ratio into the multi-agent collaborative learning system. In this system, each local execution network acts as an independent agent and makes decisions based on local observation information and the status of neighboring nodes. The dynamic coordination of local decisions is achieved through the multi-agent collaborative learning system. For example, node 1 and node 2 dynamically adjust task deployment and resource allocation by exchanging their respective load status and task execution status to achieve global optimal resource utilization.

[0099] The solution of this application can:

[0100] Improve resource utilization: By dynamically adjusting the task deployment location and resource allocation ratio, the resources of the computing nodes can be fully utilized, resource waste can be avoided, and overall resource utilization can be improved. Reduce task completion time: By deploying tasks to the most suitable computing nodes and dynamically adjusting resource allocation according to the node load, the waiting time and execution time of tasks can be reduced, thereby reducing the task completion time. Enhance system robustness: Through the multi-agent collaborative learning mechanism, local execution networks can cooperate with each other to jointly respond to emergencies, such as node failures or network congestion, thereby enhancing the robustness and stability of the system.

[0101] In an optional implementation, the hierarchical hybrid decision network is used to perform adaptive scheduling, and a distributed anomaly detection network is deployed during task execution. The distributed anomaly detection network builds a normal mode baseline of multi-dimensional performance indicators based on a variational autoencoder, monitors performance fluctuations and load balancing status in real time through an anomaly detector, and predicts resource demand trends in combination with a graph time series neural network, including:

[0102] The hierarchical hybrid decision network is used to perform adaptive scheduling, and the hierarchical hybrid decision network formulates an initial task deployment plan and a resource allocation strategy according to the real-time status information of the computing nodes as a benchmark reference for subsequent anomaly detection;

[0103] Based on the initial task deployment plan, a distributed anomaly detection network is deployed during the task execution process. The distributed anomaly detection network collects and standardizes the performance indicator data of each node in real time, including processor usage, memory occupancy, network throughput, and task response time, and generates a performance indicator vector for anomaly detection;

[0104] Input the performance indicator vector into a variational autoencoder, and construct a normal mode baseline of the multi-dimensional performance indicator through the variational autoencoder, wherein the encoder maps the performance indicator vector to a latent variable space to generate probability distribution parameters, and the decoder reconstructs the sampled latent variables back to the original feature space to establish a benchmark model for the normal operation of the system;

[0105] Based on the benchmark model, the anomaly detector continuously monitors performance fluctuations and load balancing status, calculates the deviation of real-time performance indicators from the normal mode baseline, and combines the load distribution status evaluation to generate a state description vector containing the degree of anomaly and load information; the state description vector is used as input and combined with the graph time series neural network to predict resource demand trends.

[0106] A resource adaptive scheduling method based on hierarchical hybrid decision network and distributed anomaly detection is designed to optimize resource utilization and task execution efficiency in distributed computing environments. This method integrates hierarchical decision-making, anomaly detection and prediction mechanisms, and can dynamically adjust resource allocation strategies according to the system operation status to improve system stability and robustness.

[0107] First, a hierarchical hybrid decision network is constructed. This network combines expert knowledge and machine learning algorithms to formulate initial task deployment plans and resource allocation strategies based on the real-time status information of computing nodes, such as CPU utilization, memory usage, network bandwidth, and task queue length. For example, for a cluster of 10 computing nodes, the network formulates an initial plan based on the CPU and memory capacity of the nodes, allocating computing-intensive tasks to nodes with rich CPU resources and memory-intensive tasks to nodes with rich memory resources. This initial plan will serve as a benchmark for subsequent anomaly detection.

[0108] Next, after the task starts executing, a distributed anomaly detection network is deployed. The network is distributedly deployed on each computing node to collect the performance indicator data of the node in real time. The collected data includes CPU usage, memory occupancy, network throughput, and task response time. In order to facilitate subsequent processing, the collected data is standardized, for example, using the Z-score standardization method to convert data of different dimensions into standardized data with a mean of 0 and a standard deviation of 1. The standardized data is combined into a performance indicator vector for anomaly detection. For example, if the standardized CPU usage of a node is 0.8, the memory occupancy is 0.5, the network throughput is 0.9, and the task response time is 0.2, then the performance indicator vector of the node is [0.8, 0.5, 0.9, 0.2].

[0109] Then, a variational autoencoder is used to construct a normal mode baseline for multi-dimensional performance indicators. The collected performance indicator vector is input into the variational autoencoder. The encoder maps the high-dimensional performance indicator vector to a low-dimensional latent variable space and generates the probability distribution parameters of the latent variable, such as the mean and variance. The decoder reconstructs the latent variables sampled from the probability distribution back to the original performance indicator vector. By training the variational autoencoder, it can accurately reconstruct the performance indicator vector under normal operating conditions, thereby establishing a baseline model for normal operation of the system. For example, the trained variational autoencoder can reconstruct the input [0.8, 0.5, 0.9, 0.2] into [0.78, 0.52, 0.88, 0.21], which is very close to the original vector, indicating that the system is in a normal state.

[0110] Based on the established baseline model, the anomaly detector continuously monitors performance fluctuations and load balancing status. The anomaly detector calculates the deviation of real-time performance indicators from the normal mode baseline, such as reconstruction error. At the same time, combined with the load distribution state evaluation, such as calculating the load variance of the node, a state description vector containing the degree of anomaly and load information is generated. For example, if the reconstruction error of a node is large and the load of the node is much higher than that of other nodes, the generated state description vector may be [0.5, 0.8], where 0.5 represents the degree of anomaly and 0.8 represents the degree of load imbalance.

[0111] Finally, the state description vector is used as input and combined with the graph time series neural network to predict the resource demand trend. The graph time series neural network can capture the topological relationship and time dependency between computing nodes, and predict future resource requirements based on the current system state. For example, based on the current state description vector [0.5, 0.8] and historical data, the graph time series neural network predicts that the CPU usage of the node will continue to increase in the future, and CPU resource allocation needs to be increased. The hierarchical hybrid decision network dynamically adjusts the task deployment plan and resource allocation strategy based on the prediction results, such as migrating some tasks to nodes with lower loads, or increasing the CPU resource quota of the node.

[0112] The solution of this application can:

[0113] Improve resource utilization: Avoid resource waste and improve resource utilization efficiency by dynamically adjusting resource allocation strategies. Enhance system stability: Discover and handle potential problems in a timely manner through real-time anomaly detection and prediction mechanisms to ensure stable system operation. Optimize task execution efficiency: Assign tasks to the most appropriate computing nodes through adaptive scheduling, shorten task completion time, and improve execution efficiency.

[0114] In an optional implementation, when an anomaly is detected, an adaptive tuning mechanism is triggered, the adaptive tuning mechanism uses a reinforcement transfer learning method to generate a task migration strategy and a resource reallocation plan, and realizes system optimization through progressive task migration and elastic resource adjustment. The optimization result is used to update the multimodal spatiotemporal feature perception network through knowledge distillation, and the knowledge accumulation and model evolution are realized, including:

[0115] A multimodal spatiotemporal feature perception network is used to monitor the status of computing nodes in real time and generate system anomaly detection results; when the multimodal spatiotemporal feature perception network detects an anomaly, the anomaly detection result is input into an adaptive tuning mechanism, and the adaptive tuning mechanism constructs a state space and an action space, wherein the state space includes anomaly detection results and real-time performance indicators, and the action space includes optional operations of task migration and resource allocation;

[0116] The adaptive tuning mechanism constructs a tuning network based on the reinforcement transfer learning method. The tuning network includes a main strategy network and a target strategy network. The state space information is input into the main strategy network to generate an action probability distribution. The action value is evaluated through the target strategy network, and the strategy knowledge in the historical optimization experience is transferred to the current scene. The task migration strategy and resource reallocation plan are output;

[0117] According to the task migration strategy, the task migration is performed in a progressive manner, the resource status of the target node is evaluated and a migration checkpoint is established, the incremental data transmission is performed according to the planned migration sequence, the running status of the task after the migration is verified and the system topology information is updated;

[0118] According to the resource reallocation plan, perform elastic resource adjustment, dynamically adjust the number of processor cores, memory capacity limit and network bandwidth allocation of the migrated task, and generate optimized resource configuration parameters;

[0119] Feedback the optimization results of task migration and resource adjustment through knowledge distillation, build a teacher network to save optimization experience, use the student network to learn the latest strategy, guide knowledge migration by calculating feature similarity loss function, and integrate historical experience and new knowledge;

[0120] The fused knowledge is updated to the multimodal spatiotemporal feature perception network to optimize the feature extraction and recognition capabilities of the anomaly detection model, improve the accuracy of subsequent anomaly detection, and form a closed-loop evolutionary mechanism of detection-tuning-optimization.

[0121] An adaptive system optimization method based on multimodal spatiotemporal feature perception network and reinforcement transfer learning is used to monitor and dynamically adjust computing nodes in real time to deal with system anomalies and improve overall performance. The core of this method is to build a closed-loop evolutionary mechanism of detection-tuning-optimization.

[0122] First, a multimodal spatiotemporal feature perception network is constructed. The network integrates multiple data sources, such as CPU utilization, memory occupancy, network traffic, task execution time, etc., and converts these data into multimodal feature vectors. These feature vectors contain time dimension information, such as the trend of feature changes over a period of time, and spatial dimension information, such as the feature correlation between different computing nodes. For example, a sudden increase in the CPU utilization of a node accompanied by a surge in network traffic may indicate that the node is abnormal. The perception network uses pre-trained deep learning models, such as long short-term memory networks (LSTM) or graph neural networks (GNN), to analyze these multimodal spatiotemporal features and output anomaly detection results, such as "node 3 is overloaded" or "network congestion".

[0123] When the multimodal spatiotemporal feature perception network detects an anomaly, the adaptive tuning mechanism is triggered. The mechanism first constructs a state space and an action space. The state space contains anomaly detection results and real-time performance indicators, such as "node 3 is overloaded" and the CPU utilization and memory occupancy of node 3. The action space contains optional operations for task migration and resource allocation, such as migrating task A from node 3 to node 2, or increasing the memory capacity of node 2. Assuming that the system has 3 nodes and 2 tasks, the state space can be represented as a vector containing the anomaly type, the CPU and memory usage of the 3 nodes, such as ["node 3 is overloaded", 0.9, 0.8, 0.2, 0.7, 0.3, 0.5]. The action space can be represented as a set of all possible migration combinations and resource adjustment schemes, such as {"migrate task 1 from node 3 to node 2 and increase the memory of node 2", "migrate task 2 from node 1 to node 3 and reduce the CPU core of node 1"}.

[0124] The core of the adaptive tuning mechanism is the tuning network based on reinforcement transfer learning. The network consists of a main policy network and a target policy network. The state space information is input into the main policy network to generate an action probability distribution. For example, if node 3 is overloaded, the probability of migrating tasks from node 3 to other nodes will be higher. The target policy network evaluates the value of different actions, such as the degree of improvement in the overall performance of the system after migrating tasks. The key to reinforcement transfer learning is to transfer the policy knowledge from historical optimization experience to the current scenario. For example, if migrating tasks to node 2 has a better effect in similar overload situations in the past, then in the current scenario, the probability of migrating tasks to node 2 will also be higher. Finally, the tuning network outputs the task migration strategy and resource reallocation plan. Assuming that the main policy network outputs a probability of 0.8 to migrate task 1 from node 3 to node 2 and a probability of 0.2 to migrate task 2 from node 1 to node 3, then task 1 will be finally chosen to migrate from node 3 to node 2.

[0125] According to the generated strategy, the system performs progressive task migration. First, the resource status of the target node (for example, node 2) is evaluated and a migration checkpoint is established. Then, incremental data transmission is performed according to the planned migration sequence to avoid network congestion caused by migrating a large amount of data at one time. For example, the data of task 1 is divided into multiple small blocks and gradually transferred to node 2. Finally, the running status of the task after migration is verified and the system topology information is updated to ensure that the task runs normally on the new node.

[0126] At the same time, the system performs elastic resource adjustments according to the resource reallocation plan. For example, the number of processor cores, memory capacity limit, and network bandwidth allocation of the migrated task are dynamically adjusted to generate optimized resource configuration parameters. Assume that the initial number of CPU cores of node 2 is 4 and the memory is 8GB. After migrating task 1, the system adjusts its CPU core number to 6 and memory to 10GB.

[0127] The optimization results of task migration and resource adjustment are fed back to the multimodal spatiotemporal feature perception network through knowledge distillation, thereby achieving knowledge accumulation and model evolution. Specifically, a teacher network is constructed to save optimization experience, and the student network is used to learn the latest strategy. The knowledge migration is guided by calculating the feature similarity loss function, integrating historical experience and new knowledge. Finally, the fused knowledge is updated to the multimodal spatiotemporal feature perception network to optimize the feature extraction and recognition capabilities of the anomaly detection model, improve the accuracy of subsequent anomaly detection, and form a closed-loop evolutionary mechanism of detection-tuning-optimization. For example, if it is learned through knowledge distillation that the strategy of migrating tasks to node 2 is more effective when node 2 has sufficient resources, then in similar situations in the future, the multimodal spatiotemporal feature perception network will be more inclined to identify node 2 as a suitable migration target.

[0128] The solution of this application can:

[0129] Improve system stability: This method can monitor the system status in real time and respond to anomalies in a timely manner to avoid system crashes or performance degradation, thereby improving system stability and reliability. Optimize resource utilization: By dynamically adjusting task migration and resource allocation, this method can effectively balance the load of computing nodes, avoid resource waste, and improve resource utilization. Achieve adaptive optimization: This method uses reinforcement transfer learning and knowledge distillation technology to continuously learn and accumulate optimization experience, achieve adaptive optimization of the system, and continuously improve performance over time.

[0130] In an optional implementation, the optimization results of task migration and resource adjustment are fed back through knowledge distillation, a teacher network is constructed to save optimization experience, the student network is used to learn the latest strategy, and knowledge migration is guided by calculating the feature similarity loss function. The integration of historical experience and new knowledge includes:

[0131] Collecting task migration data and resource adjustment data in the computing system, extracting performance differences before and after migration, resource consumption changes, and task completion time from the task migration data, extracting resource allocation plans, performance improvement degrees, and system load status from the resource adjustment data, and characterizing the extracted data to generate an optimization result vector;

[0132] Constructing a teacher network for storing optimization experience, wherein the feature mapping layer of the teacher network receives the optimization result vector, uses a multi-layer perceptron to extract feature associations, highlights key information through an attention mechanism, and uses residual connections to retain original features. The experience encoding layer of the teacher network uses a recurrent unit and a memory module to store the processed optimization experience;

[0133] The teacher network organizes the stored optimization experience into a mapping structure of scenario-strategy-effect, establishes a decision tree index optimization path, uses incremental learning to update the optimization experience, eliminates inefficient experience based on a value evaluation mechanism, maintains the correlation between experiences, and forms an optimization experience knowledge base;

[0134] Construct a student network for learning new scenario strategies. The student network uses a lightweight convolutional layer to extract real-time input features, selects key features through a pooling layer and an attention mechanism, and the strategy generation module of the student network generates candidate solutions based on the extracted features and uses an evaluation network to predict the execution effect.

[0135] Calculate the similarity between the optimized experience features output by the teacher network and the strategy features generated by the student network to obtain feature similarity; construct a knowledge distillation loss function based on the feature similarity, including soft label cross entropy loss and strategy consistency constraint, and adjust the degree of knowledge transfer through a temperature parameter;

[0136] Using course learning to transfer knowledge, selecting training samples based on the output of the knowledge distillation loss function, constructing a learning sequence with increasing difficulty, establishing a knowledge graph to guide the migration path, and selectively inheriting optimization experience after evaluating knowledge relevance;

[0137] The new optimization strategy learned by the student network is fed back to the optimization experience knowledge base, and the stored optimization experience is updated through the experience encoding layer of the teacher network to achieve dynamic fusion of historical experience and new scene knowledge.

[0138] A task migration and resource adjustment optimization method based on knowledge distillation. The specific implementation steps are as follows:

[0139] First, collect task migration data and resource adjustment data in the computing system. For example, record the CPU utilization, memory occupancy, network bandwidth usage, and task completion time of task A before and after it is migrated to server B; record the percentage of database query speed improvement and overall system load changes after server C adjusts the number of CPU cores. Preprocess these raw data, such as normalization and standardization, and then extract key features. For example, extract the performance difference before and after migration (for example, throughput increased by 10%), resource consumption changes (for example, memory usage decreased by 20%), and task completion time changes (for example, shortened by 5 seconds) from the task migration data. Extract the resource allocation plan (for example, adjust the number of CPU cores from 4 cores to 8 cores), performance improvement (for example, database query speed increased by 30%), and system load state changes (for example, system average load increased from 0.8 to 1.2) from the resource adjustment data. Convert the extracted features into numerical vectors to form an optimization result vector, such as [0.1, -0.2, -5, 2, 0.3, 0.4].

[0140] Next, a teacher network is constructed to store optimization experience. The feature mapping layer of the teacher network receives the optimization result vector as input. This layer uses a multi-layer perceptron to extract the correlation between features, for example, the relationship between performance improvement and resource consumption reduction. At the same time, the attention mechanism is used to highlight key information. For example, for some tasks, the change in completion time is more important than the change in resource consumption. In addition, residual connections are used to retain the original feature information to prevent information loss during the multi-layer perceptron processing. The experience encoding layer of the teacher network uses recurrent units (for example, LSTM) and memory modules to store the processed optimization experience. The recurrent unit can capture the temporal relationship in the optimization result vector, and the memory module can store long-term dependencies. For example, a resource adjustment strategy has performed well in the past month, and this information will be stored in the memory module. The teacher network organizes the stored optimization experience into a "scenario-strategy-effect" mapping structure. For example, "increasing the number of CPU cores in a high-load scenario can improve performance." A decision tree index optimization path is established to facilitate fast retrieval and matching. At the same time, the teacher network uses incremental learning to update the optimization experience. For example, when a new optimization result vector is input, the teacher network will be updated according to its similarity with the existing experience. In addition, the teacher network also eliminates inefficient experiences based on the value evaluation mechanism. For example, if a strategy does not work well after multiple attempts, it will be eliminated. Finally, the teacher network maintains the association between experiences. For example, "increasing memory" and "increasing the number of CPU cores" are often used together to form an optimization experience knowledge base.

[0141] Then, a student network is constructed to learn strategies in new scenarios. The student network uses a lightweight convolutional layer to extract real-time input features, such as the CPU utilization, memory occupancy, and network bandwidth usage of the current system. Key features are selected through pooling layers and attention mechanisms. The strategy generation module of the student network generates candidate solutions based on the extracted features, such as increasing the number of CPU cores, increasing memory, migrating tasks, etc. The evaluation network is used to predict the execution effect of each candidate solution, such as the degree of performance improvement, changes in resource consumption, etc.

[0142] Next, the similarity between the optimized experience features output by the teacher network and the policy features generated by the student network is calculated. For example, the cosine similarity between the two feature vectors is calculated. A knowledge distillation loss function is constructed based on feature similarity. The loss function contains soft label cross entropy loss and policy consistency constraint. The soft label cross entropy loss is used to measure the difference between the policy generated by the student network and the optimized experience provided by the teacher network. The policy consistency constraint is used to encourage the policy generated by the student network to be consistent with the experience provided by the teacher network. The degree of knowledge transfer is adjusted by the temperature parameter. For example, the higher the temperature parameter, the smoother the degree of knowledge transfer.

[0143] Use curriculum learning to transfer knowledge. Select training samples based on the output of the knowledge distillation loss function and build a learning sequence with increasing difficulty. For example, learn simple scenarios first, then learn complex scenarios. Build a knowledge graph to guide the migration path, for example, associate the two strategies of "increasing memory" and "increasing the number of CPU cores". Selectively inherit optimization experience after evaluating knowledge relevance, for example, only inherit experience related to the current scenario.

[0144] Finally, the new optimization strategy learned by the student network is fed back to the optimization experience knowledge base. The stored optimization experience is updated through the experience encoding layer of the teacher network to achieve dynamic fusion of historical experience and new scene knowledge. For example, if the student network learns a new and effective resource adjustment strategy, it is added to the experience knowledge base of the teacher network.

[0145] The solution of this application can:

[0146] Improve resource utilization efficiency: By learning and migrating historical optimization experience, you can find the right resource adjustment strategy more quickly, avoid unnecessary resource waste, and thus improve resource utilization efficiency. For example, when facing a new task load, the system can quickly determine the best resource allocation plan based on historical experience to avoid excess or insufficient resources. Accelerate optimization convergence speed: With the guidance of the teacher network, the student network can learn effective optimization strategies more quickly, thereby accelerating optimization convergence speed. For example, when adjusting resources in a new scenario, the student network can learn from the experience of the teacher network to avoid repeated exploration of invalid strategies, thereby finding the best solution more quickly. Enhance system adaptability: By continuously learning new scenarios and strategies, the system can continuously accumulate optimization experience, thereby enhancing the system's adaptability. For example, as the business load changes, the system can dynamically adjust the resource allocation strategy according to the new situation to maintain the efficient operation of the system.

[0147] Figure 2-Figure 3 Schematic diagram of the first and second AI-based automatic optimization technology flow for big data distributed computing tasks in the embodiments of the present invention, such as Figure 2-Figure 3 As shown, the method also includes:

[0148] Establish task optimization standards: Use industry standards such as 10TB TeraSort as test samples on the test cluster or idle business cluster to test scenarios such as file combinations of different sizes, file quantities, and data skew, and tune the computing task parameters to achieve the theoretical optimal performance of the computing task, using various performance indicators at this time as standard values;

[0149] Collect and analyze task data: Collect the running logs of all tasks scheduled to run on the big data cluster within a period of time, calculate and analyze the data of the computing tasks during the running process, and calculate the resource application, peak usage, waste rate, performance indicators, etc. of each task; further analyze according to some dimensions;

[0150] Task problem classification and labeling: Use AI to perform cluster analysis on the preliminary analysis results of the second step, calculate the classification of the top few types of problems that cause about 80% of all resource waste and the feature library of tasks with superior performance, and improve the performance model construction. And compare the standard value in the first step, classify and label each task, such as a large number of memory requests but low peak memory usage, too many subtasks, and serious data skew in some subtasks.

[0151] Training task optimization model: According to the standard values ​​and standard task parameters in the first step, the tasks of different problem types in the second step are modified and trained in a targeted manner until the optimal performance of the problem is achieved. At this time, the corresponding parameter configuration is the expected optimal parameter configuration for the task type.

[0152] Automatically adjust parameters before task scheduling: In the preparation stage before task scheduling, the original configuration parameters of the task are overwritten according to the optimal parameters of the problem type corresponding to the task mark and the training model, and submitted to the scheduling system and big data cluster for operation.

[0153] The solution of this application can:

[0154] The method provided by the embodiment of the present invention can bring about reliable improvement of task performance and automated tuning of big data distributed computing tasks, ensuring that the tasks are continuously in the optimal performance operating state without human input, saving enterprises more than 35% of computing power resources at the current cluster scale, and bringing huge benefits in reducing costs and increasing efficiency of big data clusters.

[0155] Figure 4 FIG. 1 is a schematic diagram of the structure of an automatic optimization system for big data distributed computing tasks based on AI according to an embodiment of the present invention. Figure 4 As shown, the system comprises:

[0156] The first unit is used to construct a multimodal spatiotemporal feature perception network, wherein the multimodal spatiotemporal feature perception network uses a hierarchical bidirectional long short-term memory unit to construct a temporal feature extraction layer, and converts the task arrival time, task completion time limit, data dependency, and computing load intensity in the temporal feature extraction layer into a multidimensional temporal feature sequence; a graph attention network is used to construct a spatial feature extraction layer, and the geographical distribution, network topology, computing power, storage capacity, and bandwidth resources of the computing nodes in the spatial feature extraction layer are converted into dynamic graph features; the multidimensional temporal feature sequence and the dynamic graph features are adaptively fused through a dynamic weight self-attention mechanism to generate a heterogeneous feature tensor with spatiotemporal correlation, and the expression ability of the heterogeneous feature tensor is enhanced through a contrastive learning method;

[0157] The second unit is used to train a hierarchical hybrid decision network based on the heterogeneous feature tensor, wherein the hierarchical hybrid decision network includes a global policy network and a local execution network, wherein the global policy network adopts a hierarchical deep reinforcement learning architecture, extracts hierarchical features of task queue status, resource utilization status and system load status through a transform encoder, and generates a global decision for task scheduling and resource allocation in combination with a Monte Carlo tree search and a priority experience replay mechanism; the local execution network adopts a graph neural network structure, optimizes the task deployment location and resource allocation ratio in a local computing node cluster based on the global decision, and realizes dynamic coordination of local decisions through a multi-agent collaborative learning mechanism;

[0158] The third unit is used to perform adaptive scheduling using the hierarchical hybrid decision network, deploy a distributed anomaly detection network during task execution, the distributed anomaly detection network builds a normal mode baseline of multi-dimensional performance indicators based on a variational autoencoder, monitors performance fluctuations and load balancing status in real time through an anomaly detector, and predicts resource demand trends in combination with a graph time series neural network. When an anomaly is detected, an adaptive tuning mechanism is triggered, and the adaptive tuning mechanism uses a reinforcement transfer learning method to generate task migration strategies and resource reallocation plans, and achieves system optimization through progressive task migration and elastic resource adjustment. The optimization results are used to update the multimodal spatiotemporal feature perception network through knowledge distillation to achieve knowledge accumulation and model evolution.

[0159] According to a third aspect of the embodiments of the present invention,

[0160] An electronic device is provided, comprising:

[0161] processor;

[0162] a memory for storing processor-executable instructions;

[0163] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0164] A fourth aspect of the embodiments of the present invention is:

[0165] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0166] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An AI-based automatic optimization method for big data distributed computing tasks, characterized in that: include: Constructing a multimodal spatiotemporal feature perception network, wherein the multimodal spatiotemporal feature perception network uses a hierarchical bidirectional long short-term memory unit to construct a temporal feature extraction layer, and converts the task arrival time, task completion time limit, data dependency, and computing load intensity in the temporal feature extraction layer into a multidimensional temporal feature sequence; A spatial feature extraction layer is constructed using a graph attention network, and the geographical distribution, network topology, computing power, storage capacity, and bandwidth resources of computing nodes in the spatial feature extraction layer are converted into dynamic graph features; Adaptively fuse the multidimensional temporal feature sequence and the dynamic graph feature through a dynamic weight self-attention mechanism to generate a heterogeneous feature tensor with spatiotemporal correlation, and enhance the expression ability of the heterogeneous feature tensor through a contrastive learning method; Based on the heterogeneous feature tensor training, a hierarchical hybrid decision network is constructed, wherein the hierarchical hybrid decision network includes a global policy network and a local execution network, wherein the global policy network adopts a hierarchical deep reinforcement learning architecture, extracts hierarchical features of task queue status, resource utilization status, and system load status through a transform encoder, and generates a global decision for task scheduling and resource allocation in combination with a Monte Carlo tree search and a priority experience replay mechanism; the local execution network adopts a graph neural network structure, optimizes the task deployment location and resource allocation ratio in a local computing node cluster based on the global decision, and realizes dynamic coordination of local decisions through a multi-agent collaborative learning mechanism; The hierarchical hybrid decision network is used to perform adaptive scheduling, and a distributed anomaly detection network is deployed during the task execution process. The distributed anomaly detection network builds a normal mode baseline of multi-dimensional performance indicators based on a variational autoencoder, monitors performance fluctuations and load balancing status in real time through an anomaly detector, and predicts resource demand trends in combination with a graph time series neural network. When an anomaly is detected, an adaptive tuning mechanism is triggered. The adaptive tuning mechanism uses a reinforcement transfer learning method to generate task migration strategies and resource reallocation plans, and achieves system optimization through progressive task migration and elastic resource adjustment. The optimization results are used to update the multimodal spatiotemporal feature perception network through knowledge distillation to achieve knowledge accumulation and model evolution. The graph time series neural network can capture the topological relationship and time dependency between computing nodes, and predict future resource requirements based on the current system status.

2. The method according to claim 1, characterized in that Based on the heterogeneous feature tensor training, a hierarchical hybrid decision network is constructed, wherein the hierarchical hybrid decision network includes a global policy network and a local execution network, wherein the global policy network adopts a hierarchical deep reinforcement learning architecture, extracts hierarchical features of task queue status, resource utilization status, and system load status through a transform encoder, and combines Monte Carlo tree search and priority experience playback mechanism to generate global decisions for task scheduling and resource allocation, including: Constructing a hierarchical hybrid decision network, wherein the hierarchical hybrid decision network includes a global policy network and a local execution network, wherein the global policy network adopts a hierarchical deep reinforcement learning architecture; inputting the heterogeneous feature tensor into a transform encoder, and extracting hierarchical features through a multi-head self-attention mechanism of the transform encoder, wherein the hierarchical features are used to describe the multi-level association among the task queue state, the resource utilization state, and the system load state; Inputting the hierarchical features into the global policy network, generating global decisions for task scheduling and resource allocation based on the hierarchical features; constructing a Monte Carlo tree search, using the global decision as an initial strategy to guide the search process, and obtaining a candidate decision sequence through tree search expansion; recording a state evaluation value at a tree search node, and selecting an optimal decision branch based on the state evaluation value; Design a priority experience playback mechanism to record decision trajectories and system feedback information; calculate the importance weight of the decision trajectory based on the temporal difference error and construct a priority sampling probability distribution; sample high-value training data from the experience pool according to the priority sampling probability distribution, and use the high-value training data to optimize the global strategy network; The decision output of the global policy network is sent to the local execution network, and the local execution network generates specific scheduling instructions and resource allocation solutions.

3. The method according to claim 1, characterized in that The local execution network adopts a graph neural network structure, optimizes the task deployment location and resource allocation ratio in the local computing node cluster based on the global decision, and realizes dynamic coordination of local decisions through a multi-agent collaborative learning mechanism, including: Receiving a resource allocation plan of the global decision network, wherein the resource allocation plan includes a target area for task deployment, a reference ratio for resource allocation, and constraints for load balancing; Under the guidance of the resource allocation scheme, a graph neural network structure of a local computing node cluster is constructed, wherein the nodes in the graph neural network structure include computing resource capabilities, current load status, and task execution records, and the connection edges between nodes include network bandwidth, transmission delay, and connection stability; Based on the graph neural network structure, a graph neural network is used to extract the state features of the local computing node cluster, and the node information is aggregated through the graph convolution layer and the attention layer to generate a feature vector reflecting the node computing capacity and load distribution; According to the feature vector, a matching score between the task to be deployed and the computing node is calculated, wherein the matching score comprehensively considers the remaining resources of the node and the quality of the network connection to generate a priority ranking of the task deployment; Based on the priority sorting, the task deployment location is optimized, and the resource allocation ratio of each task on the target computing node is determined through iterative adjustment, where the resource allocation ratio includes CPU ratio, memory ratio, and bandwidth ratio; The task deployment location and the resource allocation ratio are input into a multi-agent collaborative learning system. Each local execution network in the multi-agent collaborative learning system acts as an independent agent and makes decisions based on local observation information and the status of neighboring nodes. Dynamic coordination of local decisions is achieved through the multi-agent collaborative learning system.

4. The method according to claim 1, characterized in that: The hierarchical hybrid decision network is used to perform adaptive scheduling, and a distributed anomaly detection network is deployed during task execution. The distributed anomaly detection network builds a normal mode baseline of multi-dimensional performance indicators based on a variational autoencoder, monitors performance fluctuations and load balancing status in real time through an anomaly detector, and predicts resource demand trends in combination with a graph time series neural network, including: The hierarchical hybrid decision network is used to perform adaptive scheduling, and the hierarchical hybrid decision network formulates an initial task deployment plan and a resource allocation strategy according to the real-time status information of the computing nodes as a benchmark reference for subsequent anomaly detection; Based on the initial task deployment plan, a distributed anomaly detection network is deployed during the task execution process. The distributed anomaly detection network collects and standardizes the performance indicator data of each node in real time, including processor usage, memory occupancy, network throughput, and task response time, and generates a performance indicator vector for anomaly detection; Input the performance indicator vector into a variational autoencoder, and construct a normal mode baseline of the multi-dimensional performance indicator through the variational autoencoder, wherein the encoder maps the performance indicator vector to a latent variable space to generate probability distribution parameters, and the decoder reconstructs the sampled latent variables back to the original feature space to establish a benchmark model for the normal operation of the system; Based on the benchmark model, the anomaly detector continuously monitors performance fluctuations and load balancing status, calculates the deviation of real-time performance indicators from the normal mode baseline, and combines the load distribution status evaluation to generate a state description vector containing the degree of anomaly and load information; the state description vector is used as input and combined with the graph time series neural network to predict resource demand trends.

5. The method according to claim 1, characterized in that When an anomaly is detected, the adaptive tuning mechanism is triggered. The adaptive tuning mechanism uses the reinforcement transfer learning method to generate task migration strategies and resource reallocation plans, and realizes system optimization through progressive task migration and elastic resource adjustment. The optimization results are used to update the multimodal spatiotemporal feature perception network through knowledge distillation to realize knowledge accumulation and model evolution, including: A multimodal spatiotemporal feature perception network is used to monitor the status of computing nodes in real time and generate system anomaly detection results; when the multimodal spatiotemporal feature perception network detects an anomaly, the anomaly detection result is input into an adaptive tuning mechanism, and the adaptive tuning mechanism constructs a state space and an action space, wherein the state space includes anomaly detection results and real-time performance indicators, and the action space includes optional operations of task migration and resource allocation; The adaptive tuning mechanism constructs a tuning network based on the reinforcement transfer learning method. The tuning network includes a main strategy network and a target strategy network. The state space information is input into the main strategy network to generate an action probability distribution. The action value is evaluated through the target strategy network, and the strategy knowledge in the historical optimization experience is transferred to the current scene. The task migration strategy and resource reallocation plan are output; According to the task migration strategy, the task migration is performed in a progressive manner, the resource status of the target node is evaluated and a migration checkpoint is established, the incremental data transmission is performed according to the planned migration sequence, the running status of the task after the migration is verified and the system topology information is updated; According to the resource reallocation plan, perform elastic resource adjustment, dynamically adjust the number of processor cores, memory capacity limit and network bandwidth allocation of the migrated task, and generate optimized resource configuration parameters; Feedback the optimization results of task migration and resource adjustment through knowledge distillation, build a teacher network to save optimization experience, use the student network to learn the latest strategy, guide knowledge migration by calculating feature similarity loss function, and integrate historical experience and new knowledge; The fused knowledge is updated to the multimodal spatiotemporal feature perception network to optimize the feature extraction and recognition capabilities of the anomaly detection model, improve the accuracy of subsequent anomaly detection, and form a closed-loop evolutionary mechanism of detection-tuning-optimization.

6. The method according to claim 5, characterized in that Feedback the optimization results of task migration and resource adjustment through knowledge distillation, build a teacher network to save optimization experience, use the student network to learn the latest strategy, guide knowledge migration by calculating the feature similarity loss function, and integrate historical experience and new knowledge, including: Collecting task migration data and resource adjustment data in the computing system, extracting performance differences before and after migration, resource consumption changes, and task completion time from the task migration data, extracting resource allocation plans, performance improvement degrees, and system load status from the resource adjustment data, and characterizing the extracted data to generate an optimization result vector; Constructing a teacher network for storing optimization experience, wherein the feature mapping layer of the teacher network receives the optimization result vector, uses a multi-layer perceptron to extract feature associations, highlights key information through an attention mechanism, and uses residual connections to retain original features. The experience encoding layer of the teacher network uses a recurrent unit and a memory module to store the processed optimization experience; The teacher network organizes the stored optimization experience into a mapping structure of scenario-strategy-effect, establishes a decision tree index optimization path, uses incremental learning to update the optimization experience, eliminates inefficient experience based on a value evaluation mechanism, maintains the correlation between experiences, and forms an optimization experience knowledge base; Construct a student network for learning new scenario strategies. The student network uses a lightweight convolutional layer to extract real-time input features, selects key features through a pooling layer and an attention mechanism, and the strategy generation module of the student network generates candidate solutions based on the extracted features, and uses an evaluation network to predict the execution effect. Calculate the similarity between the optimized experience features output by the teacher network and the strategy features generated by the student network to obtain feature similarity; construct a knowledge distillation loss function based on the feature similarity, including soft label cross entropy loss and strategy consistency constraint, and adjust the degree of knowledge transfer through a temperature parameter; Using course learning to transfer knowledge, selecting training samples based on the output of the knowledge distillation loss function, constructing a learning sequence with increasing difficulty, establishing a knowledge graph to guide the migration path, and selectively inheriting optimization experience after evaluating knowledge relevance; The new optimization strategy learned by the student network is fed back to the optimization experience knowledge base, and the stored optimization experience is updated through the experience encoding layer of the teacher network to achieve dynamic fusion of historical experience and new scene knowledge.

7. An AI-based automatic optimization system for big data distributed computing tasks, used to implement the method described in any one of claims 1 to 6, characterized in that: include: The first unit is used to construct a multimodal spatiotemporal feature perception network, which uses a hierarchical bidirectional long short-term memory unit to construct a time series feature extraction layer, and converts the task arrival time, task completion time limit, data dependency, and computing load intensity in the time series feature extraction layer into a multidimensional time series feature sequence; A spatial feature extraction layer is constructed using a graph attention network, and the geographical distribution, network topology, computing power, storage capacity, and bandwidth resources of computing nodes in the spatial feature extraction layer are converted into dynamic graph features; Adaptively fuse the multidimensional temporal feature sequence and the dynamic graph feature through a dynamic weight self-attention mechanism to generate a heterogeneous feature tensor with spatiotemporal correlation, and enhance the expression ability of the heterogeneous feature tensor through a contrastive learning method; The second unit is used to train a hierarchical hybrid decision network based on the heterogeneous feature tensor, wherein the hierarchical hybrid decision network includes a global policy network and a local execution network, wherein the global policy network adopts a hierarchical deep reinforcement learning architecture, extracts hierarchical features of task queue status, resource utilization status and system load status through a transform encoder, and generates a global decision for task scheduling and resource allocation in combination with a Monte Carlo tree search and a priority experience replay mechanism; the local execution network adopts a graph neural network structure, optimizes the task deployment location and resource allocation ratio in a local computing node cluster based on the global decision, and realizes dynamic coordination of local decisions through a multi-agent collaborative learning mechanism; The third unit is used to perform adaptive scheduling using the hierarchical hybrid decision network, deploy a distributed anomaly detection network during task execution, the distributed anomaly detection network builds a normal mode baseline of multi-dimensional performance indicators based on a variational autoencoder, monitors performance fluctuations and load balancing status in real time through an anomaly detector, and predicts resource demand trends in combination with a graph time series neural network. When an anomaly is detected, an adaptive tuning mechanism is triggered, and the adaptive tuning mechanism uses a reinforcement transfer learning method to generate task migration strategies and resource reallocation plans, and achieves system optimization through progressive task migration and elastic resource adjustment. The optimization results are used to update the multimodal spatiotemporal feature perception network through knowledge distillation to achieve knowledge accumulation and model evolution.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images